KR102110478B1 - Headset type user interface system and user feature classify method - Google Patents

Abstract

The present invention provides a headset-type user interface system including: a near-infrared ray emission unit for emitting near-infrared rays to skin at a predetermined distance; a signal frame reception unit for receiving at least one signal frame including a signal of a near-infrared ray, which is scattered within the skin and emitted out of the skin among near-infrared rays incident on the skin by the emitted near-infrared rays; a signal frame preprocessing unit for preprocessing the at least one signal frame in an order in which the at least one signal frame is received; a feature information extraction unit for extracting feature information of the skin deformed by the emitted near-infrared rays from the preprocessed at least one signal frame; and a feature information classification unit for classifying the extracted feature information according to a preset user intention. Accordingly, hands-free natural user interface (NUI) technology for augmented reality (AR)/virtual reality (VR) is implemented.

Description

Headset type user interface system and user feature information classification method therefor {Headset type user interface system and user feature classify method}

The present invention relates to a user interface, and more specifically, a user interface in the form of a headset that implements an interface technology by grasping the user's intention through spatially resolved diffuse reflection (SRDR) according to scattering phenomenon by near infrared rays. A system and a method for classifying user feature information therefor.

Augmented reality (AR) has become one of the hottest issues in the ICT industry. In the AR field, developing a UI (user interface) device optimized for an AR headset is one of the main tasks. This is because it is difficult to use the existing UI (keyboard, mouse, touch screen, etc.) because of the headset environment that is significantly different from a personal computer or mobile phone. Pressing the button on the side of the headset or holding the remote control is the most representative solution currently adopted in the headset UI.

However, these UIs have limitations in supporting complex and dangerous operations such as surgery and manual work, and are not convenient enough for general use. Speech recognition techniques and physiological sensors such as EOG (Electrooculography), EEG (Electrencephalography), and EMG (Electromyography) are promising technologies for AR hands-free UI development, but an optimal headset UI implementation method using them has not yet been developed.

Academia and industry have already developed a technology that enables user interaction with a headset type. In general interface technology, there are various methods such as gaze recognition, hand gesture recognition, EMG sensor, touch pad, remote controller, etc. Among them, hand gesture detection or gaze tracking technology is most frequently used.

However, the interaction technology of most commercial AR headsets still does not have sufficient technology necessary for effective UI, and thus relies on the hand-held controller or button technology corresponding to the peripheral device of the headset.

This technique has limitations to be applied to special situations in which a user has to use his or her hands, such as surgery or a factory site.

Accordingly, in the UI technology, there is a need to develop a technology capable of reading a user's intention in a hands-free manner and performing commands accordingly.

Korean Patent Publication No. 2019-0056833

Accordingly, the present invention has been proposed in consideration of the above, and aims to implement hands-free NUI (AR/VR) technology for AR/VR by detecting skin deformation of a face gesture and using it as a UI input command. Is done.

In addition, the present invention aims to support hands-free interaction between a headset user and an AR system by easily detecting non-invasively moving facial skin and mapping detected facial motions to input commands.

In addition, the present invention aims to enable application in a special situation or environment by implementing a headset type user interface technology.

In addition, an object of the present invention is to receive data related to skin deformation in a two-dimensional form, so that quantitative and qualitatively improved data can be received.

In addition, an object of the present invention is to facilitate data classification by generating data related to a change in a user's motion through calculating the amount of change of continuously received data.

In addition, an object of the present invention is to implement feature extraction and classification by an unsupervised learning method, so that a learning dataset can be classified without tagging information.

In addition, the present invention aims to improve the accuracy problem, which is a limitation of an unsupervised learning method, by applying a DEC (Deep Embedded Clustering) technique.

The objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, the user interface system in the form of a headset according to the technical concept of the present invention is a near-infrared emitting unit that emits near-infrared light to the skin at a certain distance, and among the near-infrared rays incident on the skin by the emitted near-infrared rays. A signal frame receiving unit for receiving at least one signal frame that contains signals of near infrared rays scattered in the skin and emitted out of the skin, a signal frame pre-processing unit for pre-processing the at least one signal frame in a received order, and the pre-processed at least It may include a feature information extraction unit for extracting feature information of the skin deformed by the emitted near-infrared from one or more signal frames, and a feature information classification unit for classifying the extracted feature information into a predetermined user's intention.

The near-infrared emission unit may emit collimated near-infrared rays so that reception of a signal frame according to scattering phenomena caused by near-infrared rays emitted to the skin minimizes the influence of reflection phenomena caused by the near-infrared rays emitted to the skin.

The signal frame received by the signal frame receiver may be in the form of a near infrared image including spatially resolved diffuse reflectance (SRDR) measured from scattering caused by near infrared rays emitted to the skin.

The signal frame receiving unit may include a near-infrared filter so that the near-infrared image does not include noise signals of other bands in addition to the near-infrared signal of the target band.

The signal frame pre-processing unit may pre-process the near-infrared images in the order of reception, and reconstruct the data format according to the data input format of the feature extraction unit.

The feature information extracting unit may extract at least one feature for classifying the user's intention from the pre-processed near-infrared image, and the feature information classification unit may classify and map the extracted at least one feature to the user's intention.

The feature information extractor extracts the feature information using an encoder of a pre-trained spatiotemporal autoencoder (STAE) based on a deep neural network (DNN), and the feature information classifier is DEC (Deep Embedded) Based on the clustering technique, the extracted at least one feature may be classified using a pre-trained classifier and mapped to a preset user intention.

In order to achieve the above object, the method for classifying user feature information for a user interface according to the technical idea of the present invention includes a near-infrared emission step of emitting near-infrared rays to the skin at a certain distance from the near-infrared emission unit, and the emitted from the signal frame receiving unit A signal frame receiving step of receiving at least one signal frame including signals of near infrared rays scattered within the skin and emitted out of the skin among the near infrared rays incident on the skin by the near infrared rays, wherein the at least one signal is received by the signal frame preprocessing unit A signal frame pre-processing step of pre-processing the frames in the received order, a feature information extraction step of extracting feature information of the skin deformed by the emitted near infrared rays from the at least one signal frame pre-processed by the feature information extraction unit, and feature information The classification unit may include a feature information classification step of classifying the extracted feature information into a preset user's intention.

The near-infrared emission step may emit collimated near-infrared rays so that the reception of a signal frame according to the scattering phenomenon caused by the near-infrared rays emitted to the skin minimizes the influence of the reflection phenomenon caused by the near-infrared rays emitted to the skin.

In the signal frame receiving step, the received signal frame may be in the form of a near infrared image including spatially resolved diffuse reflection (SRDR) measured from scattering caused by near infrared rays emitted to the skin.

The signal frame receiving unit may include a near-infrared filter so that the near-infrared image does not include noise signals of other bands in addition to the near-infrared signal of the target band.

In the signal frame pre-processing step, the near-infrared image may be pre-processed in the order of reception, and the data form may be reconstructed according to the data input form of the feature extraction step.

The feature information extraction step extracts at least one feature for classifying the user's intention from the pre-processed near-infrared image, and the feature information classification step classifies and maps the extracted at least one feature to the user's intention. have.

The feature information extraction step extracts the feature information using an encoder of a pre-trained spatiotemporal autoencoder (STAE) based on a deep neural network (DNN), and the feature information classification step is DEC ( Based on the Deep Embedded Clustering technique, the extracted at least one feature may be classified using a pre-trained classifier and mapped to a preset user intention.

According to the user interface system in the form of a headset as described above and a method for classifying user feature information therefor,

First, after detecting the skin deformation of the face gesture, it has the effect of realizing hands-free NUI (NUI) technology for AR/VR by using it as a UI input command.

Second, it has the effect of easily detecting a non-invasive movement of the face skin and mapping the detected face motion to an input command to support hands-free interaction between the headset user and the AR system.

Third, by implementing a user interface technology in the form of a headset, it has an effect that can be applied in special situations or environments.

Fourth, by receiving the data on the skin deformation in a two-dimensional form, it has the effect of receiving quantitative and qualitatively improved data.

Fifth, by generating the data related to the change of the user's motion through the calculation of the amount of change of continuously received data, data classification has an easy effect.

Sixth, by implementing feature extraction and classification as an unsupervised learning method, it has an effect of classifying a learning dataset without tagging information.

Seventh, through the application of the DEC (Deep Embedded Clustering) technique, it has the effect of improving the accuracy problem, which is a limitation of the unsupervised learning method.

1 is a block diagram showing a user interface system in the form of a headset according to an embodiment of the present invention.
2 is a flowchart illustrating a method for classifying user feature information for a user interface according to an embodiment of the present invention.
3 is a view showing a hands-free UI system according to an embodiment of the present invention.
4 is a view showing the principle of the infrared camera according to FIG. 3;
Figure 5 is implemented in accordance with an embodiment of the present invention, showing a sensor module applied to the AR headset.
6 is an embodiment of the present invention, a diagram showing some IR diffusion patterns recorded by the sensor module while the user is wearing an AR headset.
7 is a view showing a pre-processing process for a clustering network as an embodiment of the present invention.
8 is a diagram illustrating a network structure for extracting sensor data features as an embodiment of the present invention.
9 is a diagram showing details of a network structure of a STAE used for feature extraction as an embodiment of the present invention.
FIG. 10 is a diagram illustrating a classifier network composed of a feature extractor based on a STAE and a feature classifier based on a DEC, as an embodiment of the present invention.
11 is a diagram illustrating clustering results by applying a fine adjustment technique (DEC) as an embodiment of the invention.
12 is a view showing a screenshot demonstrated using a custom application, as an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings and the contents described in the accompanying drawings, which illustrate preferred embodiments of the present invention. Features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. Prior to this, the terms or words used in the present specification and claims are based on the principle that the inventor can appropriately define the concept of terms in order to explain the best way of his own invention. It should be interpreted as a matching meaning and concept. In addition, it should be noted that detailed descriptions of known functions and configurations related to the present invention are omitted when it is determined that the subject matter of the present invention may be unnecessarily obscured.

1 is a block diagram showing a user interface system in the form of a headset according to an embodiment of the present invention.

Referring to Figure 1, the user interface system in the form of a headset according to an embodiment of the present invention is largely near-infrared emission unit 100, signal frame receiving unit 200, signal frame pre-processing unit 300, feature information extraction unit 400 , It may include a feature information classification unit 500.

The near-infrared emission unit 100 may emit near-infrared light at a predetermined distance. This can be said to be a component for sensing the skin in a non-contact manner. At this time, the near-infrared emitting unit 100 may be referred to as a laser diode, not an LED. The reason is that the signal can be recognized by using the LED in contact with the skin, whereas the laser diode can recognize the signal without touching the skin.

In this case, the near-infrared emission unit 100 emits collimated near-infrared rays so that reception of a signal frame according to scattering phenomena caused by near-infrared rays emitted to the skin minimizes the effect of reflection caused by the near-infrared rays emitted to the skin. Can be. The reason is that signal scattering or measurement may not be possible because the infrared scattering phenomenon caused by the near infrared rays emitted from the skin is blocked by the infrared reflection phenomenon caused by the near infrared rays emitted by the skin. Accordingly, the near-infrared emitting unit 100 may emit sufficiently collimated near-infrared rays to the skin.

The signal frame receiving unit 200 includes at least one signal frame including signals of near infrared rays scattered within the skin and emitted from the skin among the near infrared rays incident on the skin by the near infrared rays emitted from the near infrared emission unit 100. I can receive it. This can be said to be a component for receiving a signal regarding skin deformation.

In this case, the signal frame receiving unit 200 may be referred to as an infrared camera, not an infrared photodiode (PD). The reason is that the infrared photodiode acquires one-dimensional point data, while the infrared camera can acquire two-dimensional data using camera image data. This can be said to be quantitative and qualitatively improved data.

The signal frame received from the signal frame receiving unit 200 may be in the form of a near infrared image including spatially resolved diffuse reflection (SRDR) measured from scattering phenomena caused by near infrared rays emitted to the skin. This may be a characteristic for grasping the user's intention through spatially resolved diffuse reflection (SRDR) according to scattering phenomenon caused by near infrared rays.

Meanwhile, the signal frame receiver 200 may include a near-infrared filter so that the near-infrared image does not include noise signals in other bands in addition to the near-infrared signal in the target band.

The signal frame pre-processing unit 300 may pre-process at least one signal frame in a received order. This can be said to be a component for improving the performance of the feature information extraction and classification from the received signal frame. At this time, the reason for pre-ordering the received order is that it is possible to confirm the change diffused in the skin by the emitted near infrared rays.

In this case, the signal frame pre-processing unit 300 may reconstruct the data type according to the data input form of the following feature information extraction unit 400.

The feature information extracting unit 400 may extract feature information of the skin deformed by the emitted near infrared rays from at least one signal frame pre-processed by the signal frame pre-processing unit 300.

In more detail, the feature information extraction unit 400 may extract at least one feature for classifying the user's intention from the near-infrared image pre-processed by the signal frame pre-processing unit 300.

At this time, the feature information extracting unit 400 may extract the feature information using an encoder of a spatiotemporal autoencoder (STAE) pre-trained based on a deep neural network (DNN).

The feature information classifying unit 500 may classify feature information extracted from the feature government extracting unit 400 to a predetermined user's intention.

In more detail, the feature information classifying unit 500 may classify and map at least one feature extracted from the feature information extracting unit 400 to a user's intention.

At this time, the feature information classifying unit 500 may classify the extracted at least one feature using a pre-trained classifier based on a DEC (Deep Embedded Clustering) technique and map it to a preset user intention.

The feature information extracting unit 400 and the feature information classifying unit 500 are based on an unsupervised learning method, which may have a feature of classifying a learning dataset without tagging information. Among them, the DEC (Deep Embedded Clustering) technique can be said to improve the accuracy problem, which is a limitation of the unsupervised learning method.

2 is a flowchart illustrating a method for classifying user feature information for a user interface according to an embodiment of the present invention.

Referring to FIG. 2, a method for classifying user feature information for a user interface according to an embodiment of the present invention is largely a near infrared emission step (S100), a signal frame receiving step (S200), a signal frame preprocessing step (S300), and feature information extraction It may include a step (S400), a feature information classification step (S500).

In the near-infrared emission step, the near-infrared emission unit 100 may emit near-infrared light at a predetermined distance from the skin (S100). This can be said to be a step for sensing the skin in a non-contact manner. At this time, in the near-infrared emission step (S100), the near-infrared emission unit 100 may be referred to as a laser diode, not an LED. The reason is that the signal can be recognized by using the LED in contact with the skin, whereas the laser diode can recognize the signal without touching the skin.

In this case, the near-infrared emission step (S100) emits collimated near-infrared rays so that reception of a signal frame according to scattering phenomena caused by the near-infrared rays emitted to the skin minimizes the effect of reflection caused by the near-infrared rays emitted to the skin. Can be. The reason is that signal scattering or measurement may not be possible because the infrared scattering phenomenon caused by the near infrared rays emitted from the skin is blocked by the infrared reflection phenomenon caused by the near infrared rays emitted by the skin. Accordingly, the near-infrared emission step (S100) may emit sufficiently collimated near-infrared rays to the skin.

The signal frame receiving step receives at least one signal frame including signals of near infrared rays scattered within the skin and emitted out of the skin among the near infrared rays incident on the skin by the near infrared rays emitted from the near infrared emission step (S100). It can be (S200). This may be a step for receiving a signal regarding skin deformation.

At this time, in the signal frame receiving step S200, the signal frame receiving unit 200 may be an infrared camera, not an infrared photodiode (PD). The reason is that the infrared photodiode acquires one-dimensional point data, while the infrared camera can acquire two-dimensional data using camera image data. This can be said to be quantitative and qualitatively improved data.

The signal frame received in the signal frame receiving step S200 may be in the form of a near infrared image including spatially resolved diffuse reflection (SRDR) measured from scattering phenomena caused by near infrared rays emitted to the skin. This may be a characteristic for grasping the user's intention through spatially resolved diffuse reflection (SRDR) according to scattering phenomenon caused by near infrared rays.

Meanwhile, the signal frame receiver 200 may include a near-infrared filter so that the near-infrared image does not include noise signals in other bands in addition to the near-infrared signal in the target band.

In the signal frame pre-processing step, at least one or more signal frames may be pre-processed in the received order (S300). This can be said to be a step for improving the performance of feature information extraction and classification from the received signal frame. At this time, the reason for pre-ordering the received order is that it is possible to confirm the change diffused in the skin by the emitted near infrared rays.

At this time, the signal frame pre-processing step S300 may reconstruct the data form according to the data input form of the following feature information extraction step S400.

The feature information extraction step may extract feature information of the skin deformed by the emitted near-infrared from at least one signal frame pre-processed from the signal frame pre-processing step (S300) (S400).

In more detail, the feature information extraction step (S400) may extract at least one feature for classifying the user's intention from the near-infrared image preprocessed from the signal frame pre-processing step (S300 ).

At this time, the feature information extraction step S400 may extract the feature information using an encoder of a spatiotemporal autoencoder (STAE) pre-trained based on a deep neural network (DNN).

In the feature information classification step, feature information extracted from the feature government extraction step (S400) may be classified according to a preset user's intention (S500).

In more detail, the feature information classification step (S500) may classify and map at least one feature extracted from the feature information extraction step (S400) according to a user's intention.

At this time, the feature information classification step (S500) may classify the extracted at least one feature using a pre-trained classifier based on a DEC (Deep Embedded Clustering) technique and map it to a preset user intention.

The feature information extraction step (S400) and the feature information classification step (S500) are based on an unsupervised learning method, which may have a feature that a learning dataset can be classified without tagging information. Among them, the DEC (Deep Embedded Clustering) technique can be said to improve the accuracy problem, which is a limitation of the unsupervised learning method.

Referring to the present invention in more detail, reference may be made to FIGS. 3 to 12.

FIG. 3 is a view showing a hands-free UI system according to an embodiment of the present invention, and FIG. 4 is a view showing a shooting principle of the infrared camera according to FIG. 3.

5 is implemented according to an embodiment of the present invention, and may be referred to as a diagram showing a sensor module applied to an AR headset.

Referring first to FIGS. 3 and 4, in a hands-free UI system according to an embodiment of the present invention, a sensor module may include an IR LD and an IR camera. The IR camera captures (captures) an image of the IR diffusion pattern, and the IR LD is a component that emits IR light to the skin.

In Figure 5 (a) is a sensor module (Sensor module) may include a USB camera and a NIR laser diode. This can be said to be installed on the left side of the Epson BT-350 AR glasses. (b) is a user wearing a headset, and a laser diode may target the skin near the left cheek. The skin near the left cheek may deform when the user makes a wink gesture.

Deformation of human skin can be detected by measuring the change in IR SRDR delivered under the skin (in the skin). At this time, SR (Spatially resolved) is a diffuse reflection measured according to a radial distance from the point of incidence of light, and diffuse reflectance (DR) is a scattering plane that is a ratio of the amount of flux re-emitted by the diffuse reflection. You can disappear. The strength of DR can be influenced by the microstructure of collagen fibers distributed under the skin.

According to an embodiment of the present invention, an embodiment in which a user's expression is sensed using an IR diffuse reflector can be implemented based on that the randger's line representing collagen fiber distribution in human skin follows the SRDR measurement direction of the skin. have.

Based on this, the embodiments of FIGS. 3 to 5 will be described in more detail as follows.

Referring to FIG. 3, in a hands-free UI system according to an embodiment of the present invention, a sensor module that detects an IR diffusion pattern from a skin surface of a sensor wearer and a sensor data clustering based on a DNN structure Module).

The sensor module can be mounted on a wearable AR headset. The sensor module can consist of a pair of LD and USB cameras. These can be called IR emitters and receivers, respectively. While the user is wearing AR glasses, IR light emitted from the LD can diffuse through the user's skin. The camera may take a sequential image of the diffused IR light and transmit it to a classifier to convert the user's skin deformation state into an interface command.

In this case, the AR headset manufactured through one embodiment may use a commercial AR headset (Epson BT-350) and a face gesture tracking sensor module. In addition, the face gesture tracking sensor module is mounted on the left side of the AR glasses as shown in FIG. 5, and the ROI sensor detects a point between the user's left eye and ear where the skin is locally stretched or compressed when the user makes a wink gesture. It can be said that it was a goal.

The NIR LD (ie, IR emitter) had a center wavelength of 850 nm with an average output voltage of 0.4 mW, which can be classified as a level 1 laser. The NIR illumination did not interfere with the user's field of view, and it can be said that the NIR dose was limited to a safe level to prevent potential harmful effects on the user's eyes.

And the band of the NIR can penetrate the deepest skin of the harmless light spectrum. This is an important characteristic in selecting a light source for sensor implementation, along with low power consumption. The reason for choosing LD instead of light emitting diode (LED) is that the IR light from the LED is more durable than it actually is, but in contrast to the LD light, the IR light from the LED can be transmitted through the user's skin when it is not in direct contact with the skin. This is because collimation is insufficient.

The IR emitter may include an LD, a protection circuit, and a collimation lens. The protection circuit can extend the life of the LD, and the collimating lens can collimate the IR laser beam to reduce the dispersion of the emission. The IR receiver is a sensor module that includes an OV2710 chipset camera. In this case, the sensor module may incorporate an IR band pass filter capable of detecting IR light without interference from ambient visible light. The field of view (FOV) of the camera is 100 degrees, and can be recorded at 120 fps with 320 X 240 resolution according to the manufacturer's specifications. The camera in one embodiment of the present invention can be run at 20 fps (on average) due to data processing latency. In fact, a 320 x 240 resolution Grayscale jpeg image was used for the input data.

FIG. 6 is an embodiment of the present invention, and may be a view showing some IR diffusion patterns recorded by the sensor module while the user is wearing the AR headset.

In more detail, in FIG. 6, (a) may be referred to as IR SRDR photographed without a face gesture. (b) may be referred to as IR SRDR photographed while the user makes a wink gesture.

Referring to FIG. 6, the NIR band band pass filter installed in the camera can cut all wavelength bands except 850 nm (NIR). The shape of the pattern can be changed in relation to the user's face gesture.

The input data input to the sensor data clustering module may be included in the form of an NIR SRDR image of a user's skin. Although the SRDR shape photographed by the camera is the same as the user's face gesture, there may be a person-to-person deformation due to various factors that rarely model the face gesture. In an embodiment of the present invention, the sensor data classifier must detect deformation of the SRDR shape photographed in order to recognize a user's gesture.

Accordingly, in the present invention, as an embodiment, an unsupervised learning, that is, an unsupervised learning method, may be used for a neural network network sensor data classifier. The unsupervised learning method can be said to be a room that can be easily corrected when it is necessary to cope with a personal-to-person transformation. This is because the unsupervised learning method may not work with data annotation to create a training data set.

The process of training the sensor data classifier can consist of the following two steps.

(Step 1) Features can be extracted from the data set through unsupervised learning using an autoencoder.

(Step 2) The performance of the classifier can be improved by clustering and fine-tuning the extracted features.

As shown in FIG. 5, in an embodiment of the present invention, training data for a classifier neural network network may be collected using a built-in IR sensor module. More specifically, 5,000 or more sequential SRDR images per person can be collected without data annotation on the training data set for the classifier network. Here, 81,758 SRDR images collected from 10 different users can be used to train the classifier network.

The data input to the classifier network is a series of four 320 X 240 pixel IR diffuse snapshot images, which can be taken sequentially by the camera and stored in an input image queue. The classifier network may perform pre-processing before sending the image to the feature extraction portion of the network.

7 is an embodiment of the present invention, and may be referred to as a diagram showing a pre-processing process for a clustering network.

More specifically, the pre-processing unit in FIG. 7 can calculate a difference between two images. After calculation, the pre-processing unit can set the input image as thresholds, perform subtraction by pixel between the two images, and cluster it by adjusting it to a size of 28 X 28 pixels to fit the clustering network input size. It can be grouped into groups as long as the input length required by the network.

As shown in FIG. 7, the pre-processing unit can emphasize the deformation of the SRDR contour shape and then group the four sequential images according to the size of the RNN window of the STAE to create a classifier network by focusing on the shape variation of the SRDR contour. have.

In addition, the pre-processing unit may first convert the input image to a black and white binary image having a predetermined threshold to emphasize the contour type of each SRDR. The pre-processing unit can then calculate the pixel-by-pixel difference of the SRDR binary image between regular time intervals to emphasize the deformation of the SRDR contour. At this time, the constant time interval can be referred to as two sampling intervals between the current sampling time [n] and the previous sampling time [n-2]. Afterwards, to reduce the computational process, the SRDR binary image can be reduced to a 28 X 28pixel image size. Images before shrinking can be stored in the image queue. Finally, in order to input the pre-processing unit to the feature extraction portion, four sequential images can be synthesized as a set for the input image sequence.

The image at the end of the input sequence ([n]th) can be said to be the most recent image taken from the camera.

8 is a diagram illustrating a network structure for extracting sensor data features as an embodiment of the present invention.

In more detail, in FIG. 8, the STAE is composed of an encoder and a decoder, and only the encoder part can be used for feature extraction after learning of the STAE.

As shown in FIG. 8, the sensor camera of an embodiment of the present invention may convert a user's face gesture into a sequential NIR SRDR image. The converted images were data input to the classifier network, which can be said to be data that can determine whether or not to imply a wink gesture. Image clustering is an optimal method for classifying images in an unsupervised learning method. However, this method still has limitations for the following two reasons.

The first reason is the dimension of the input image data. Images often have very high dimensionality, which should be reduced to avoid the curse of dimensionality, which greatly affects the performance of the K-means algorithm.

The second reason is that the features in the image usually have a 2D or 3D local structure, which should not be ignored during clustering. These features may require kernels optimized for the shape of the local structure. Furthermore, since the data input to the sensor is sequential data having temporal characteristics, the clustering network of the present invention must be able to extract temporal characteristics and spatial information from the input.

In an embodiment of the present invention, STAE can be adopted to reduce the input order while preserving the spatiotemporal characteristics of the input without user supervision. The encoder part of the autoencoder can make a feature extraction part by a clustering method. This is because the encoder can learn to compress input data while minimizing compression loss while training the autoencoder.

를 중심

STAE으로 대표되는 k 클러스터에 클러스터링한 경우, 비선형 매핑

을 가진 각

의 차원 수를 줄일 수 있다. 여기서

은 커널의 중량이고, Z은 잠재된 특징 공간이라 할 수 있다. 커널은 차원 감소를 지원할 뿐만 아니라, 이미지 시퀀스에서 특징을 감지할 수 있는 용량을 가질 수 있다. STAE 학습이 끝나면, 학습

은 완료되고(이는 분류기 파라미터의 초기화를 의미할 수 있다.), STAE의 디코더 부분은 더 이상 필요하지 않을 수 있다. 그런 다음 후속 부스팅 단계에서 폐기될 수 있다.STAE can be said to be one of the types of autoencoder composed of recurrent neural network (RNN) and convolutional neural network (CNN) for extracting temporal and spatial features from input data, respectively. n image sequence

Centered

When clustering on a k cluster represented by STAE, nonlinear mapping

Angle with

Can reduce the number of dimensions. here

Is the weight of the kernel, and Z is the potential feature space. The kernel not only supports dimensionality reduction, but also has the capacity to detect features in an image sequence. When STAE learning is over, learn

Is completed (which may mean initialization of the classifier parameters), and the decoder portion of the STAE may no longer be needed. It can then be discarded in a subsequent boosting step.

Autoencoder is a kind of artificial neural network that learns data representation using an unsupervised learning method and is generally a part of an encoder-decoder pair. The encoder is trained to compress the input data, and the decoder can be trained to restore the compressed data to the original input data with minimal loss. Here, the encoder may be induced to learn a data representation to reduce unnecessary redundancy in the input data.

8, the STAE in one embodiment of the present invention can receive four consecutive images as input data from the pre-processing unit.

9 is a diagram showing details of a network structure of a STAE used for feature extraction as an embodiment of the present invention.

As shown in FIG. 9, in order to find spatial features and reduce the size of the input image, two 3D CNN layers may be disposed in the STAE network with the highest priority. To reduce the number of dimensions of the input image, each convolutional layer of the encoder can operate in a non-padding mode. To initialize the kernel weights, you can use a distributed scale. Each CNN layer can be followed by a maximum pooling layer that cuts the image size in half. At this time, since the input data is 3D (width, height, sampling time), the network needs to process the input data using a 3D CNN layer rather than a typical 2D CNN.

Then, two convolutional convolutional long-short-term memory (LSTM) layers can extract temporary features from the compressed input data.

Conventional RNNs can learn transient features through hidden state parameters, but they have a vanishing gradients problem. LSTM can alleviate the slope loss problem by introducing a repetitive gate called a forget gate. ConvLSTM is a type of LSTM designed to find time information from a sequence of image frames such as image data, and may have a 2D shape weight. Instead of using multiplication operations between inputs and weights, ConvLSTM can use convolution operations between inputs. The LSTM layer may have a function of propagating spatiotemporal features through each LSTM state.

Finally, in one embodiment of the present invention, another 3D CNN layer is arranged after the ConvLSTM layer designed to have the same kernel size as the compressed input image, and 4 2D images that can suggest the extracted features of the 4 2D images. Can be converted to vectors.

The arranged 3D CNN layer may map the output of the previous layer to 2D space for clustering. In the present invention, the dimension of the feature space can be preset to match the intended number of clusters. Due to the number of clusters that only want to include two states (skin deformation state and no deformation state), the dimension of the feature space can be set to 2D in the present invention. Of the four extracted vectors, only the last vector containing the feature of the most recently photographed data can be used as the final result of feature extraction in the sensor data clustering process.

The decoder can be configured completely symmetrically with the configuration of the encoder, except for adopting the average unpooling layer instead of using the maximum unpooling layer for the convenience of implementing the network. After learning about the autoencoder, the decoder can be discarded in the classifier network because it is no longer needed. Only the encoder part can be used in the feature extractor.

10 is a diagram illustrating a classifier network composed of a feature extractor based on a STAE and a feature classifier based on a DEC, as an embodiment of the present invention.

As illustrated in FIG. 10, clustering is one of unsupervised learning methods, which can be used to group data having similar characteristics. Compared with the classifier based on the supervised learning method, the classifier based on the unsupervised learning method may have an advantage that it is not necessary to provide a background answer to the ambiguous classification target data.

Meanwhile, in the present invention, as an embodiment, a k-average algorithm may be adopted to implement clustering. The reason is that the k-average algorithm is simple and relatively efficient compared to other clustering algorithms. However, the k-average algorithm based on unsupervised clustering is generally somewhat less accurate than the supervised classification method.

The sensor in the embodiment of the present invention may adopt a deep embedded clustering (DEC) method to enhance clustering network performance. The clustering method in the present invention may include the following two steps to improve clustering quality.

(Step 1) The first is to find the initial cluster center by applying the K-means clustering algorithm to the sensor data feature space.

(Step 2) The second is to increase the accuracy by repeatedly applying a fine-tuning technique (DEC) to the entire classifier network until it provides more effects.

와, Z에 입력을 위한 인코드 데이터에 대한 STAE의 가중치

를 동시에 조정함으로써 데이터를 클러스터 할 수 있다.The fine-tuning technique (DEC) is a set of pre-trained k cluster centers in feature space Z

And, the weight of the STAE for the encoded data for input to Z

Data can be clustered by simultaneously adjusting.

와 인코더

의 가중치는 미세 조정 기법(DEC) 과정 동안 파라미터가 최적의 클러스터링 결과를 보여줄 때까지 반복적으로 계산될 수 있다. Deep embedded clustering(DEC)의 첫 번째 단계로서, K-평균 클러스터링 알고리즘을 이용하여 원래의 분포 Q를 구하고 Q의 클러스터링 결과가 보다 잘 구별되게 하는 목표 분포 P의 소프트 할당(soft assignment)을 계산할 수 있다. 여기에서 목표 분포 P의 클러스터간 분산을 기존 분포 Q의 클러스터간 분산보다 크게 하도록 스튜던트 T분포(Student’s T-distribution)를 소프트 할당의 커널 함수로써 사용할 수 있다. 그런 다음, KL 분산을 이용하여 상기 두 분포의 간의 차이를 구할 수 있다. 여기서 KL 분산은 보통 상기 두 분포 간의 차이를 구하기 위해 사용될 수 있다. 상기 계산된 차이는 네트워크 훈련에 대한 Deep embedded clustering(DEC)의 손실함수로 사용될 수 있다. 이는 손실을 역전파(backpropagation)하고,

및

을 업데이트함으로써 분류기의 성능을 향상시키기 위한 것이라 할 수 있다.center

And encoder

The weight of can be repeatedly calculated until the parameter shows an optimal clustering result during the DEC process. As a first step in deep embedded clustering (DEC), the K-means clustering algorithm can be used to obtain the original distribution Q and calculate the soft assignment of the target distribution P that makes the clustering results of Q better distinguishable. . Here, the Student's T-distribution can be used as a kernel function of soft allocation so that the inter-cluster distribution of the target distribution P is larger than the inter-cluster distribution of the existing distribution Q. Then, the difference between the two distributions can be obtained using KL variance. Here, the KL variance can usually be used to find the difference between the two distributions. The calculated difference can be used as a loss function of deep embedded clustering (DEC) for network training. This backpropagates the loss,

And

It can be said to improve the performance of the classifier by updating.

의 하드 할당(hard assignment)이 기 설정된 임계값 내에서 변하지 않을 때까지 반복될 수 있다. 미세 조정 기법 과정 이후, 클러스터의 인코더 가중치와 중심 좌표는 입력 이미지 스트림의 분류에 대해 최적화될 수 있다.This process

It can be repeated until the hard assignment of (hard assignment) does not change within a preset threshold. After the fine tuning process, the encoder weights and center coordinates of the cluster can be optimized for the classification of the input image stream.

11 is a diagram illustrating clustering results by applying a fine adjustment technique (DEC) as an embodiment of the invention.

More specifically, in FIG. 11, (a) is a diagram showing clustering results for 81,758 training data set images, and (b) is a diagram showing clustering results sensed in real time from a user (on-line validation).

As shown in Figure 11, the classifier network clustered a sequence of 81,758 training set input images to verify the performance of the clustering process. The clustering result can be represented as a 2D feature space using the t-SNE method. The color and shape of the dots represent each cluster in a neutral or wink state, and the name of the cluster can be represented in the legend of the scatter plots. Compared to the results obtained before applying the fine-tuning technique (DEC), the results after applying the fine-tuning technique (DEC) greatly improved the resolution of the cluster boundary as learning began.

However, in the present invention, the accuracy of a clustering network cannot be determined because uncommented data is used in the DEC-based classifier and feature extraction. In the present invention, since the unsupervised learning method is applied to the classifier network, there is no information indicating whether the user state is wink or neutral.

In an embodiment of the present invention, in order to evaluate the accuracy of the classifier network, an additional experiment (experiment on a second set of experimental data) can be performed on users wearing an AR headset.

At this time, since the data set for verification and testing has not been collected, the personal online sensor data can be used as the verification data set. In the experimental validation step, after applying a trained weight to the network according to the number of learning, it is possible to measure classification accuracy.

11(b) is a diagram showing real-time clustering results for online sensor data obtained through additional experiments. Each experimental participant took 30 wink gestures during each measurement, and the present invention can record the number of successful detections of this intentional wink gesture. As shown in FIG. 11( b), it can be seen that as the learning step increases, the performance of the clustering network is improved. After repeating the fine-tuning technique (DEC) learning 500 times, the classifier network can achieve 30/30 accuracy in wink gesture detection.

The gesture detection system in the present invention can be operated in a real-time mode at a speed of 30 fps. In order to test whether such a classifier network can be applied to other people who have volunteered to participate in data collection, the present invention may perform a third additional experiment shown in the screenshot from a demonstration using a customized application.

The user can pop a balloon (a,b) or select a button to change the background (c,d). The user can select an object with a red center point on a specific object, and then execute a wink gesture.

In the third experiment, participants were able to check the gesture detection results in real time in response to facial motions, and receive 1 point when the wink gesture was detected by the sensor. Each participant needed 100 points to complete the experiment. In the present invention, the number of wink gestures intended by each participant can be counted until each experiment is completed. All of these experiments were conducted on 10 new participants (ie, those who did not participate in the training data set data collection). This can achieve an average accuracy of 95.4% as shown in Table 1 below. Table 1 may be a table showing real-time measurement results for gesture detection accuracy of a user wearing the AR headset of the present invention.

<Table 1>

The interface technology of the present invention uses an IR camera as an input device, but it can be said that it is completely different from a general image-based facial expression classification. The sensor in the present invention can detect the movement of muscles under the skin surface by tracking the distorted alignment of collagen fibers in the skin based on the IR diffuse reflector.

In particular, in the present invention, since the skin deformation of the ROI is sensed without contact with the skin, an entire face image is not required. In general, since humans use different facial muscles to blink and intentionally wink, the SRDR sensor in the present invention has an advantage of distinguishing the two motions. In addition, the advantage of this sensing method is that the sensor in the present invention can generate classification features from ordinary facial skin surfaces where no features can be found with the camera.

In the present invention, it is possible to achieve clustering of an IR spread image generated by a wink of a user wearing an AR headset through a customized face gesture detection sensor and a classifier network. That is, the sensor in the present invention can provide an efficient method of implementing a UI for an AR headset.

Many commands can be applied in combination with facial gestures and head rotation data, the latter of which can be captured by a gyro sensor integrated into most AR headsets. For example, the user can move some object in the virtual space by selecting a button through a wink, or turning a head with a wink gesture similar to a drag-and-drop command commonly used in desktop environments.

12 is a view showing a screenshot demonstrated using a custom application, as an embodiment of the present invention. In more detail, in FIG. 12, the user can pop a balloon (a,b) or select a button to change the background (c,d), select an object with a red center point as a target, and then wink You can do this.

Referring to FIG. 12, a user can pop a balloon or select a button in an AR environment through a wink gesture. This can provide a user-friendly UI with the advantages of simple hands-free control and low cost implementation.

The sensor in the present invention requires an appropriate location, and can be limited to a specific location in the human body to detect facial skin deformation. This is because the amount of change in IR SRDR is sufficient when the muscles under the skin surface are sufficiently moved to affect the internal structure of the skin.

In order to get the best results, it may be important to set the proper ROI of the sensor. This is because an error may occur if the sensor is improperly positioned. On the other hand, since the wink gesture can be a boring or even difficult gesture for some people, in the present invention, it is possible to detect various face gestures by selecting the sensor detection location as another location. Alternatively, the number of detectable face gestures can be increased by adding more sensors. This can be easily changed and added through various experiments according to the user environment or suitability.

Although described and illustrated in connection with the preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as illustrated and described, and deviates from the scope of the technical idea. It will be understood by those skilled in the art that many changes and modifications to the present invention are possible without. Accordingly, all such suitable changes and modifications should also be considered within the scope of the present invention.

100: near-infrared emission unit 200: signal frame receiving unit
300: signal frame pre-processing unit 400: feature information extraction unit
500: feature information classification

Claims (14)

In the headset-type user interface system,
A near-infrared emitting unit which emits near-infrared rays to the skin at a certain distance;
A signal frame receiver configured to receive at least one signal frame including signals of near infrared rays scattered within the skin and emitted from the skin among the near infrared rays incident on the skin by the emitted near infrared rays;
A signal frame pre-processing unit pre-processing the at least one signal frame in a received order;
A feature information extraction unit for extracting feature information of the skin deformed by the emitted near infrared rays from the at least one signal frame pre-processed; And
Includes a; feature information classification unit for classifying the extracted feature information to a predetermined user's intention,
The signal frame received by the signal frame receiving unit is in the form of a near infrared image including spatially resolved diffuse reflection (SRDR) measured from scattering caused by near infrared rays emitted to the skin,
The signal frame pre-processing unit pre-processes the near-infrared image in the order of reception, reconstructs the data format according to the data input format of the feature information extraction unit,
The feature information extracting unit extracts at least one feature for classifying the user's intention from the pre-processed near infrared image,
The feature information classifying unit classifies and maps the extracted at least one feature to the user's intention,
The feature information extraction unit extracts the feature information using an encoder of a spatiotemporal autoencoder (STAE) pre-trained based on a deep neural network (DNN),
The feature information classifier is a headset-type user interface system that maps the extracted at least one feature to a preset user intention by using a pre-trained classifier based on a DEC (Deep Embedded Clustering) technique. According to claim 1, wherein the near-infrared emitting unit,
A user interface system in the form of a headset that emits collimated near infrared rays so that reception of a signal frame according to scattering phenomenon caused by near infrared rays emitted to the skin minimizes the effect of reflection caused by the near infrared rays emitted to the skin. delete According to claim 1, wherein the signal frame receiving unit,
A headset-type user interface system including a near-infrared filter so that the near-infrared image does not include noise signals in other bands in addition to the near-infrared signal in the target band. delete delete delete A method for classifying user feature information for a user interface,
A near-infrared emission step of emitting near-infrared rays to the skin at a certain distance from the near-infrared emission unit;
A signal frame receiving step of receiving at least one signal frame including a signal of a near infrared ray scattered within the skin and emitted out of the skin among the near infrared rays incident on the skin by the near infrared ray emitted from the near infrared ray emitting step from the signal frame receiving unit ;
A signal frame preprocessing step in which the signal frame preprocessing unit preprocesses at least one signal frame received from the signal frame reception step in a received order;
A feature information extracting unit extracting feature information of the skin deformed by the emitted near infrared rays from at least one signal frame pre-processed from the signal frame pre-processing step; And
The feature information classification unit includes a feature information classification step of classifying feature information extracted from the feature information extraction step into a predetermined user's intention.
In the signal frame receiving step, the signal frame received is in the form of a near-infrared image including spatially resolved diffuse reflectance (SRDR) measured from scattering caused by near-infrared rays emitted to the skin,
In the signal frame pre-processing step, the near-infrared image is pre-processed in the order of reception, and the data form is reconstructed according to the data input form of the feature information extraction step,
The feature information extraction step extracts at least one feature for classifying the user's intention from a near-infrared image preprocessed from the signal frame pre-processing step,
The feature information classification step classifies and maps at least one feature extracted from the feature information extraction step to the user intention,
The feature information extraction step extracts the feature information using an encoder of a spatiotemporal autoencoder (STAE) pre-trained based on a deep neural network (DNN),
The feature information classification step is a user feature information classification method for a user interface that classifies the extracted at least one feature using a pre-trained classifier based on a DEC (Deep Embedded Clustering) technique and maps it to a preset user intention. The method of claim 8, wherein the near-infrared emission step,
A method for classifying user feature information for a user interface that emits collimated near infrared rays so that reception of a signal frame according to scattering phenomena caused by the near infrared rays emitted to the skin minimizes the effect of reflection caused by the near infrared rays emitted to the skin . delete The method of claim 8, wherein the signal frame receiving unit,
A method of classifying user feature information for a user interface including a near-infrared filter so that the near-infrared image does not include noise signals of other bands in addition to the near-infrared signal of the target band. delete delete delete

Images