KR20190141390A - A hand-free glasses type hearing aid, a method for controlling the same, and computer recordable medium storing program to perform the method - Google Patents

Abstract

The present invention relates to a hands-free glasses type hearing aid, a method for controlling the same, and a computer readable medium storing a program to perform the method. The hands-free glasses type hearing aid capable of controlling volume comprises: a glasses frame including a rim, a bridge, a temple, and a tip; a speaker installed at the tip and outputting an audio signal; a camera unit installed in at least one of the rim, the bridge, and the temple to take an eye region image, which is an image of an area including eyes and eyebrows of left and right sides of a user; and a control unit controlling the volume of the audio signal outputted from the speaker according to shapes and positions of the eyes and eyebrows of the analyzed eye region image by analyzing the image photographed by the camera unit.

Description

Hands free glasses type hearing aid, a method for controlling the same, and a computer readable recording medium having recorded thereon a program for performing the method. perform the method}

The present invention relates to a spectacle hearing aid, and more particularly, to a spectacle hearing aid capable of controlling a hearing aid without a hand manipulation, a method for controlling the same, and a computer readable recording method for executing the method. It relates to a recording medium.

Recently, due to the aging society and the rapid increase in the elderly population, the elderly patients with hearing loss called 'invisible disease' is increasing. In addition, the number of patients with 'sound-induced hearing loss', a kind of disease caused by temporary and continuous exposure to loud music sounds through smartphones and earphones in the daily life, airplane noise in the city, and strong noise from construction or factories, has increased. It is a trend. Accordingly, the demand for hearing aids for the elderly and hearing loss patients is increasing rapidly as a universal welfare medical device.

Registered Korea Patent No. 1827535 Feb 02, 2018 (Name: Eyeglass detachable bone conduction hearing aid and eyeglass leg modular bone conduction and airway hearing aid)

It is an object of the present invention to control the volume by recognizing a user's desire through an image of an eye region including the user's eyes, eyebrows, or the like, to remove noise such as acoustic echo, or to increase the sensitivity of a sound in a desired direction. The present invention provides a hands-free eyeglass type hearing aid device capable of controlling the same, a method for controlling the same, and a computer-readable recording medium having recorded thereon a program for performing the method.

Hands-free eyeglass type hearing aid device according to a preferred embodiment of the present invention for achieving the above object is an eyeglass frame including a rim, a bridge, a temple and a tip, an audio unit installed on the tip portion and outputting an audio signal; A camera unit installed in at least one of the rim, the bridge, and the temple to capture an eye region image, which is an image of an area including eyes and eyebrows of the left and right sides of the user; and analyzing the image photographed by the camera unit. And a control unit for controlling the hands-free eyeglass type hearing aid according to the shape and position of the eye and eyebrow of the analyzed eye region image.

The controller may change the volume, band, and channel of the audio signal output to the audio unit according to the shape and position of the eye and eyebrow of the analyzed eye region image.

The audio unit includes an audio receiving module, which is a plurality of directional microphones installed in predetermined regions of the rim, the bridge, and the temple, and receives audio signals. The direction of orientation of the directional microphone is controlled according to the position.

The control unit includes a feature point detection module for detecting a feature point of a predetermined region of the eye and the eyebrow in an eye region image which is an image of the region including the eye and the eyebrow, a displacement measuring module for measuring the displacement of the feature point, and the measured feature point And a control generation module for generating and outputting a control signal for controlling the hands-free eyeglass type hearing aid according to the displacement.

The feature points include the left and right ends of the eyebrows, the top and bottom ends of the eyebrows, the left and right ends of the eye, the top and bottom ends of the eyes, and the pupils.

Hands-free eyeglass type hearing aid device according to a preferred embodiment of the present invention for achieving the above object is an eyeglass frame including a rim, a bridge, a temple and a tip, an audio unit installed on the tip portion and outputting an audio signal; It is installed in at least one area of the rim, bridge and temple to photograph the area including the eyes and eyebrows of the left and right sides of the user to obtain an image composed of pixel values of a plurality of pixels and a three-dimensional coordinate value of the plurality of pixels. A camera unit configured to generate an eye region image including a coordinate mask including a coordinate mask, and a controller configured to analyze the eye region image to control an eyeglass type hearing aid according to the shape and position of the eye and the eyebrow of the analyzed eye region image. .

The controller may include a plurality of layers configuring a next layer through a plurality of calculations to which the output of any one layer is weighted to determine the strength of the inter-layer connection, and when the eye region image is input, the plurality of layers An artificial neural network for performing an operation and outputting the result of the operation, a learning module for learning the artificial neural network using the eye region image and a control signal that is a target value, and the learning of the artificial neural network when the artificial neural network is completed, And a control module for inputting an image to the artificial neural network to derive an output value and to control the spectacle hearing aid using a control signal corresponding to the derived output value.

The artificial neural network may include a left neural network that outputs a median value through an operation in which weights of a plurality of layers are applied to a left image that is an image of an area including an eye and an eyebrow of a left side of the user, A weight is applied to a right neural network that outputs a median value through an operation in which weights of a plurality of layers are applied to a right image that is an image of an area including an eyebrow, and a median value of each of the left neural network and the right neural network. It includes a merged network for calculating the output value of the artificial neural network through the operation.

Each of the left neural network and the right neural network includes a plurality of convolution layers, and a plurality of kernels of at least one convolution layer among the plurality of convolution layers have the same size as an input layer, and the at least one convolution layer Each of the convolution operations using a plurality of kernels of the eyebrows, the upper part of the eye, the lower part of the eye and the eye is characterized in that extracting the features of the shape of each eye.

Each of the left neural network and the right neural network includes a plurality of convolution layers, and each of the convolution operations using a plurality of kernels of at least one convolution layer among the plurality of convolution layers is an eyebrow, an upper part of an eye, and a lower part of an eye. And extracting features of each position of the pupil.

The learning module inputs an eye region image having a predetermined control signal as the target value to the artificial neural network, and corrects the weight such that an output value of the artificial neural network has a minimum difference from the target value.

The learning module inputs a left image of the eye region image to the left neural network, and corrects the weight of the left neural network through a despreading algorithm such that a median value of the left neural network has a minimum difference from a target value. The left neural network is trained, and the right one of the eye region images is input to the right neural network, and the weight of the right neural network is obtained through a despreading algorithm such that the intermediate value of the right neural network has a minimum difference from a target value. Corrects the learning of the right neural network, inputs an eye region image including the left image and the right image to the artificial neural network, and despreads the output value of the artificial neural network to minimize the difference from the target value. Learning the artificial neural network by correcting the weight of the artificial neural network through an algorithm It shall be.

The control module may include an eye region image including the left image, which is an image of an area including the eyes and eyebrows on the left side of the user, and a right image, which is an image of the area including the eyes and eyebrows on the right side of the user. Input, derive an output value of the neural network for the eye region image, determine a control signal corresponding to the output value, control the volume of an audio signal output from the audio unit according to the determined control signal, or the audio It is characterized by changing the band and channel of the audio signal output to the negative, or to control the direction of the directional microphone.

Method for controlling the hands-free eyeglass type hearing aid device according to a preferred embodiment of the present invention for achieving the above object is to the left and right of the user through the camera unit installed in at least one of the rim, bridge and temple of the eyeglass frame Photographing an area including an eye and an eyebrow to generate an eye area image that is an image of an area including an eye and an eyebrow on the left and right sides of a user, and analyzing the image captured by the camera to analyze the image of the eye area And controlling the hands-free eyeglass type hearing aid according to the shape and position of the eye and eyebrow.

The controlling of the hands-free eyeglass type hearing aid device may include detecting a feature point of a predetermined area of the eye and the eyebrow in an eye region image, which is an image of the area including the eye and the eyebrow, measuring the displacement of the feature point, and measuring Changing the volume, band, and channel of the output audio signal according to the displaced feature point, or controlling the directing direction of the plurality of directional microphones receiving the audio signal.

The feature points include the left and right ends of the eyebrows, the top and bottom ends of the eyebrows, the left and right ends of the eye, the top and bottom ends of the eyes, and the pupils.

The method for controlling the hands-free eyeglass type hearing aid device according to the preferred embodiment of the present invention for achieving the above object is the eyes of the left and right of the user through a camera unit installed in at least one of the rim, bridge and temple. And photographing an area including the eyebrows to generate an eye area image including an image consisting of pixel values of a plurality of pixels and a coordinate mask including three-dimensional coordinate values of the plurality of pixels, and a plurality of weights applied thereto. Analyzing the eye region image by using an artificial neural network including a plurality of hierarchical layers to detect a control signal corresponding to the eye region image according to the shape and position of the eye and the eyebrow of the eye region image; And controlling the spectacle hearing aid according to the detected control signal.

The detecting of the control signal may include calculating a median value by applying a plurality of layers of weights to a left image, which is an image of an area including an eye and an eyebrow of the left side of the user, and an eye of the right side of the user. Calculating a median value through a calculation in which weights of a plurality of layers are applied to a right image, which is an image of an area including eyebrows, and a weight is applied to a median value of each of the left and right neural networks; Calculating an output value of the artificial neural network through operation; and detecting a control signal corresponding to the output value.

The calculating of the intermediate value may include extracting features of the shapes of the eyebrows, the upper part of the eye, the lower part of the eye, and the pupils through a convolution operation using a plurality of kernels.

The calculating of the median value may include extracting features of the eyebrows, the upper part of the eye, the lower part of the eye, and the positions of the pupils through a convolution operation using a plurality of kernels.

Before detecting the control signal, a left image of the eye region image is input to a left neural network of the artificial neural network, and a despreading algorithm is performed such that an intermediate value of the left neural network has a minimum difference from a target value. Training the left neural network by correcting the weight of the left neural network; and inputting a right image of the eye region image into the right neural network of the artificial neural network, and the median value of the right neural network is different from the target value. Training the right neural network by correcting a weight of the right neural network through a despreading algorithm so as to be the minimum; and inputting an eye region image including the left image and the right image to the artificial neural network, Through the despreading algorithm, the artificial neural network Training the artificial neural network by correcting weights.

The controlling of the eyeglass type hearing aid device may include controlling a volume of an audio signal output from an audio output module installed at a tip portion of the eyeglass frame according to the detected control signal, or a band and a channel of an audio signal output to the audio output module. Or changing the direction of the directional microphone installed in a predetermined region of the rim, bridge and temple of the spectacle frame.

A computer-readable recording medium having recorded thereon a program for performing a method for controlling a hands-free spectacle hearing aid according to a preferred embodiment of the present invention for achieving the above object is at least one of a rim, a bridge and a temple of a spectacle frame. Taking a region including the eyes and eyebrows of the user's left and right through a camera unit installed in the region of the user and includes a coordinate mask including an image consisting of pixel values of the plurality of pixels and three-dimensional coordinate values of the plurality of pixels. Generating an eye region image, analyzing the eye region image, detecting a control signal corresponding to the eye region image according to the shape and position of the eye and the eyebrow of the eye region image, and detecting the detected control signal And controlling the spectacle hearing aid according to the present invention.

According to the present invention, by analyzing the shape and position of the eye and eyebrows of the eye region image to generate a control signal intended to be manipulated by the user consciously through the eye, such as volume control, the hearing aid device can be operated without the hand operation. .

Furthermore, according to the present invention, the eyes and the eyebrows of the eye region image by analyzing the shape and position of the user, although unconscious or unintentional, makes an uncomfortable facial expression due to noise, or recognize the eye that the eyes are in the direction of sound The control device may generate a control signal to cancel noise or control the hearing aid device to listen to the sound in a direction in which the gaze is possible.

1 is to define the names of each part of the spectacle frame according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a hands-free eyeglass type hearing aid device according to an embodiment of the present invention.
3 is a diagram illustrating a detailed configuration of an eye region image according to an exemplary embodiment of the present invention.
4 is a block diagram illustrating a detailed configuration of a control unit according to an embodiment of the present invention.
5 is a block diagram illustrating a configuration of an artificial neural network according to an embodiment of the present invention.
6 is a view for explaining a plurality of layers of an artificial neural network according to an embodiment of the present invention.
7 is a diagram illustrating an example of a convolutional layer of an artificial neural network according to an embodiment of the present invention.
8 is a diagram illustrating an example of a convolution operation of an artificial neural network according to an embodiment of the present invention.
9 is a flowchart illustrating a method for controlling the hands-free eyeglass type hearing aid device according to the embodiment of the present invention.
10 is a flowchart illustrating a learning method of an artificial neural network according to an embodiment of the present invention.
11 is a flowchart illustrating a learning method of an artificial neural network according to another embodiment of the present invention.
12 is a flowchart illustrating a method for controlling the hands-free eyeglass type hearing aid device according to the embodiment of the present invention.
13 is a block diagram illustrating a detailed configuration of a controller according to another embodiment of the present invention.
14 is a diagram for describing a feature point in an eye region image, according to another exemplary embodiment.
15 is a flowchart illustrating a detailed configuration of a control unit according to another embodiment of the present invention.

Prior to the description of the present invention, the terms or words used in the specification and claims described below should not be construed as being limited to the ordinary or dictionary meanings, and the inventors should consider their own invention in the best way. For the purpose of explanation, it should be interpreted as meaning and concept corresponding to the technical idea of the present invention based on the principle that the concept can be properly defined as the concept of term. Therefore, the embodiments described in the present specification and the configuration shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical idea of the present invention, and various equivalents may be substituted for them at the time of the present application. It should be understood that there may be water and variations.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that like elements are denoted by like reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may blur the gist of the present invention will be omitted. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically illustrated, and the size of each component does not entirely reflect the actual size.

The spectacle-type hearing aid device according to the embodiment of the present invention basically installs a function module on the spectacle frame. Accordingly, the names of the parts of the spectacle frame are defined prior to the description of the spectacle hearing aid. 1 is to define the names of each part of the spectacle frame according to an embodiment of the present invention.

Referring to FIG. 1, the spectacle frame 10 includes a rim, a bridge, a pad, a temple, and a tip. Rim means a portion forming the edge of each of the pair of lenses. A bridge refers to a portion connecting a pair of rims. The temple means a portion that is drawn outwardly from the rim, that is, in the opposite direction of the bridge, and is connected to a predetermined portion of the rim edge through a hinge. The tip extends outward from the temple to represent the portion worn on the user's ear.

Next, a configuration of the hands-free eyeglass type hearing aid device according to the embodiment of the present invention will be described with reference to the names of the parts of the eyeglass frame described above. 2 is a block diagram illustrating a configuration of a hands-free eyeglass type hearing aid device according to an embodiment of the present invention. 3 is a diagram illustrating a detailed configuration of an eye region image according to an exemplary embodiment of the present invention.

1 and 2, the eyeglass type hearing aid device 100 according to the embodiment of the present invention includes the camera frame 110 and the audio unit 130 installed in the eyeglass frame 10 as well as the eyeglass frame 10 described above. , An input unit 140, a projection unit 150, a storage unit 160, and a control unit 170.

The camera unit 110 is for taking an image. The camera unit 110 is installed on the spectacle frame 10. For example, the camera unit 110 may be installed in at least one of a rim bridge and a temple of the spectacle frame 10. The camera unit 110 generates an eye region image by photographing an eye region of the user when the user wears it. Here, the eye area includes its peripheral area including the eye and the eyebrows. In particular, the eye region image includes a left image and a right image. The left image is an image of an area including the eyes and eyebrows of the user's left side, and the right image is an image of an area including the eyes and eyebrows of the user's right side. The camera unit 110 includes an image sensor, and the image sensor receives light reflected from a subject and converts the light into an electrical signal, and may be implemented based on a Charged Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS), or the like. Can be. The camera unit 110 may further include an analog-to-digital converter, and converts an electric signal output from an image sensor into a digital sequence to output pixel values (eg, RGB values) of each pixel. Can be. In particular, the camera unit 110 includes a 3D sensor. The 3D sensor is a sensor for obtaining three-dimensional coordinates for each pixel of the image in a non-contact manner. The camera unit 110 may capture an object and detect coordinate values (eg, x, y, and z values) of three-dimensional coordinates for each pixel of the image captured by the 3D sensor. At this time, the coordinate value of the three-dimensional coordinates is a coordinate value when the focal point of the camera unit 110 is set to 0 points. According to an embodiment of the present invention, the camera unit 110 may be two, and generates a left image and a right image, respectively. Therefore, the coordinate values of the left image and the coordinate values of the right image may be generated based on different focal points. The 3D sensor may use various types of sensors using lasers, infrared rays, visible light, and the like. These 3D sensors are laser-based three-dimensional scanners using any one of TOP (time of flight), phase-shift and online waveform analysis, laser-based three-dimensional scanners using optical triangulation, optical using white or modulated light Type 3D scanner, Handheld Real Time type PHOTO, Optical type 3D scanner, Optical type using Pattern Projection or Line Scanning, Laser type full body scanner, Photo type scanner using Photogrammetry, Kinect Fusion A real time scanner or the like used may be exemplified. As described above, the camera unit 120 according to an embodiment of the present invention generates an eye region image by capturing an eye region of the user. 3, the eye region image including the left image L and the right image R may include an image LI and RI formed of pixel values (eg, RGB values) of each pixel. ) And a coordinate mask (LM, RM) consisting of coordinate values (eg, x, y, z values) representing three-dimensional coordinates of each pixel.

The audio unit 130 includes an audio input module 131 which collects an audio signal such as a voice and transmits it to the controller 170 and an audio output module 133 for outputting an audio signal provided from the controller 170. . In an embodiment of the present invention, the audio input module 131 includes a plurality of directional microphones (MIC). That is, a plurality of microphones MIC may be installed in at least one region of the rim, the bridge, and the temple of the spectacle frame 10, and since the microphones MIC are directional microphones, the direction of the microphones is controlled according to the control of the controller 170. Can be controlled. The audio output module 133 may exemplify a speaker SPK, a bone density conductor or a bone density speaker, an earphone, or the like. In particular, the earphone may be a wired or wireless earphone. The audio output module 133 is installed at the tip portion of the spectacle frame 10 and may output audio signals of a plurality of bands and a plurality of channels. The audio output module 133 may basically output a volume of the audio signal under the control of the controller 170. In addition, the audio output module 133 may remove the noise of the audio signal by changing the output band and the output channel of the audio signal under the control of the controller 170. Such noise can typically be representative of noise caused by acoustic echo.

The input unit 140 receives a user's key operation for controlling the hearing aid device 100, generates an input signal, and transmits the generated input signal to the control unit 170. The input unit 140 may include various types of keys for controlling the hearing aid device 100. The input unit 140 according to the embodiment of the present invention may be omitted as a preliminary.

The projection unit 150 is formed on the rim or temple portion of the spectacle frame 10 and is for projecting a predetermined image onto the lens of the spectacle frame 10. When the projection unit 150 controls the state of the hearing aid device 100 itself, such as a battery state, an error message, a volume control state, a noise control state, a microphone directing state, and the like, whether or not control is performed. You can project an image that contains information about the status that shows. The projection unit 150 includes a light source, an optical system, and a projection lens. When a plurality of light sources output light constituting the image, the optical system matches the optical axes of the light output from the plurality of light sources to form light as if the light comes from one light source. Then, the light output from the optical system is magnified through the projection lens and projected onto the lens.

The storage unit 160 stores programs and data necessary for the operation of the hearing aid device 100. In particular, the storage 160 may store user data generated according to the use of the hearing aid device 100. Various data stored in the storage 160 may be deleted, changed, or added according to a user's manipulation.

The controller 170 may control an overall operation of the hearing aid device 100 and a signal flow between the internal blocks 110 to 160 of the hearing aid device 100 and may perform a data processing function for processing data. The controller 170 may be a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), or the like.

The controller 170 analyzes an eye region image captured by the camera unit 110 to determine a control signal intended by the user, and outputs the audio output module 133 of the audio unit 130 according to the determined control signal. The volume of the audio signal may be controlled, the band and channel of the output audio signal may be changed, or the directing direction of the directional microphone MIC of the audio input module 131 may be controlled. To this end, according to an embodiment, the controller 170 may include an artificial neural network 210, a learning module 220, and a control module 220. In addition, according to another exemplary embodiment, the controller 170 may include a feature point derivation module 310, a displacement measurement module 310, and a control generation module 310.

First, a method of controlling the hands-free eyeglass type hearing aid device according to an embodiment of the present invention will be described. 4 is a block diagram illustrating a detailed configuration of a control unit according to an embodiment of the present invention. Referring to FIG. 4, according to an embodiment of the present invention, as described above, the controller 170 includes an artificial neural network 210, a learning module 220, and a control module 220.

The artificial neural network 210 includes a plurality of layers. The plurality of layers constitute the next layer through a plurality of operations in which the output of any one layer is weighted to determine the strength of the inter-layer connection. When the eye region image is input, the artificial neural network 210 performs a plurality of calculations to which weights are applied, and outputs the result of the calculation.

The artificial neural network 210 may intentionally increase the volume of the audio signal output from the audio output module 133 or learn an expression of the eye region to decrease the volume through the eye region image that is the training data. The facial expression of the eye region may be, for example, blinking eyes or raising eyes.

In addition, the artificial neural network 210 may learn the facial expression of the eye region through which the user perceives the moment when the noise of the audio signal output from the audio output module 133 is outside the acceptable limit through the eye region image as the training data. . The facial expression of the eye region may be, for example, a grimace through the eyes, the eyebrows, and the muscles around it.

In addition, the artificial neural network 210 may learn an expression of an eye region in which a gaze moves in a direction in which a user wants to hear a sound through an eye region image that is training data. The facial expression of the eye region may be, for example, a movement of the pupil in a specific direction.

The learning module 220 is for learning the artificial neural network 210 as described above. Such learning uses an eye region image which is learning data and a control signal which is a target value corresponding to the eye region image. For example, if the learning data is an eye region image including an expression of an eye region for increasing or decreasing the volume of an audio signal, the target signal may be a control signal for increasing the volume or a control signal for decreasing the volume. have. In addition, if the learning data is an eye region image including an expression of an eye region for recognizing noise of an audio signal, a control signal for changing a band or channel of the corresponding audio signal may be a target value. If the learning data is an eye region image including an expression of an eye region in which a gaze moves in a direction in which a sound is generated, a control signal for controlling the directing direction of the directional microphone MIC to direct the sound direction is a target value. Can be.

The learning module 220 inputs the eye region image as the training data and the target value corresponding to the control signal to the artificial neural network 210, and despreads the output value of the artificial neural network 210 to minimize the difference between the target value and the target value. The weight of the artificial neural network 210 is corrected through a back-propagation algorithm.

When the learning of the artificial neural network 210 is completed, the control module 230 inputs an eye region image generated by the camera unit 110 to the artificial neural network 210 to derive an output value, and a control signal corresponding to the derived output value. To control the hearing aid device 100 using the.

In more detail, the control module 230 receives an eye region image which is actual data from the camera unit 110 in real time. Here, the actual data means an eye region image captured by the camera unit 110 in real time. When emphasized, the eye region image includes a left image, which is an image of an area including eyes and eyebrows of the user's left eye, and a right image, which is an image of an area including eyes and eyebrows of the user's right eye. When the control module 230 receives the eye region image that is actual data, the control module 230 inputs the received eye region image to the artificial neural network 210. Then, the neural network 210 will output an output value through a plurality of operations in which weights of a plurality of layers are applied to the eye region image.

For example, if the learning is completed, the artificial neural network 210 performs a plurality of operations to which the weight is applied when the actual data is an eye region image including an expression of an eye region for increasing or decreasing the volume of an audio signal. The height outputs a control signal or a control signal for decreasing the volume as an output value.

In addition, the artificial neural network 210 is a control region for changing the band or channel of the audio signal by performing a plurality of operations to which the weight is applied if the actual data is an eye region image including the expression of the eye region for recognizing the noise of the audio signal Will output the signal as an output value.

The artificial neural network 210 is an eye region image including an expression of an eye region in which eye gaze moves in a direction in which sound is generated, and performs a plurality of operations to which a weight is applied to the direction of the directional microphone (MIC). It will output a control signal to control the sound direction in the direction of sound.

Accordingly, the control module 230 identifies the control signal corresponding to the output value of the artificial neural network 210, and controls the volume of the audio signal output from the audio output module 133 according to the identified control signal, or audio output The band and channel of the audio signal output to the module 133 may be changed, or the direction of the directional microphone MIC may be controlled.

Next, the detailed configuration of the artificial neural network according to the embodiment of the present invention will be described. 5 is a block diagram illustrating a configuration of an artificial neural network according to an embodiment of the present invention. 6 is a view for explaining a plurality of layers of an artificial neural network according to an embodiment of the present invention. 7 is a diagram illustrating an example of a convolutional layer of an artificial neural network according to an embodiment of the present invention. 8 is a diagram illustrating an example of a convolution operation of an artificial neural network according to an embodiment of the present invention.

Referring to FIG. 5, the artificial neural network 210 according to the embodiment of the present invention includes a left neural network 211, a right neural network 213, and a merged network 215.

The left neural network 211 receives the left image L of the eye region image, calculates and outputs an intermediate value through a plurality of calculations to which weights are applied. The right neural network 213 receives the right image R of the eye region image, calculates and outputs an intermediate value through a plurality of calculations to which weights are applied. The merged network 215 receives an intermediate value output from the left neural network 211 and an intermediate value output from the right neural network 213, applies weights thereto, and calculates and outputs a final output value through a predetermined operation. do.

Next, detailed configurations of the left neural network 211 and the right neural network 213 will be described with reference to FIG. 6. Each of the left neural network 211 and the right neural network 213 includes a plurality of layers. The left neural network 211 and the right neural network 213 are both an input layer IL, a convolution layer CL, a pooling layer PL, and a fully-connected layer. FL) and a middle layer (ML).

The input layer IL is composed of a matrix having a predetermined size. Each element of the input layer IL matrix corresponds to each pixel of the image LI, RI and the coordinate mask LM, RM of the eye region image. The pixel value (e.g., RGB value) of each pixel of the image (LI, RI) and the coordinate value (e.g., x, y, z value) of each pixel of the coordinate mask (LM, RM) are the matrix of the input layer IL. It is entered as each element of.

According to an embodiment, as shown in FIG. 6, the left neural network 211 and the right neural network 213 may include a first convolutional layer CL1, a first pooling layer PL1, and a second convolutional layer CL2. ) And the second pooling layer PL2, but the present invention is not limited thereto, and the convolutional layer CL and the pooling layer PL may each exist as one or two or more pairs. Can be. Further, as shown in FIG. 6, when the convolutional layer CL and the pooling layer PL exist in two or more pairs, the convolutional layer CL and the pooling layer PL are alternated. Although illustrated as being arranged as, the present invention is not limited thereto.

Each of the convolutional layers CL: CL1 and CL2 and the pooling layers PL: PL1 and PL2 consists of a plurality of feature maps, each of which is a matrix of a predetermined size. The value of each element of the matrix constituting the feature map is calculated by applying convolution or pooling or subsampling using the kernel K to the matrix value of the previous layer. Here, the kernel K is a matrix having a predetermined size, and the value of each element of the matrix constituting the kernel K becomes a weight w.

The fully connected layer FL includes a plurality of nodes (or sigmoids: f1, f2, f3, ..., fn), and the calculation of the fully connected layer is also applied to the weight w to apply the plurality of nodes of the middle layer ML. Are input to nodes (m1, m2, m3, ... mm).

The middle layer ML may be composed of a plurality of nodes (or sigmoids: m1, m2, m3, ... mm). Each of the plurality of intermediate nodes m1, m2, m3, ... mm may correspond to a predetermined control signal. The median value, which is the output of each of the plurality of intermediate nodes m1, m2, m3, ... mm, is a probability value. For example, the first intermediate node m1 corresponds to a volume up which is a first control signal for increasing the volume of the audio signal, and a first intermediate value that is an output of the first intermediate node m1 is input. The left image or the right image represents a probability of being an eye region image (for example, an eye raised up) to perform a volume up which is a first control signal. As another example, the third intermediate node m3 corresponds to a channel change channel, which is a third control signal for changing the channel of the audio signal for noise removal, and is a third output node of the third intermediate node m3. The median value indicates the probability that the input left image or right image is an eye region image (eg, frowning) requiring a change channel, which is a third control signal.

Each of the plurality of layers IL, CL, PL, FL, ML includes a plurality of operations. Each of a plurality of operations of the plurality of layers IL, CL, PL, FL, and ML is applied with a weight w, and the weighted calculation result is transferred to the next layer. In other words, the operation result of the previous layer becomes the input of the next layer. In more detail, the operation of each layer and its weight w will be described using the example shown in FIG. 6 as an example.

For example, the input layer IL is a matrix having a predetermined size, and each element of the matrix has a pixel size. An element of the matrix of the input layer IL is a pixel unit and corresponds to each pixel of an eye region image including an image and a coordinate mask. The pixel value (e.g., RGB value) of each pixel of the image of the eye region image and the coordinate value (e.g., x, y, z value) of each pixel of the coordinate mask are binary data, and the angle of the matrix of the input layer IL It is entered in the element.

Then, a convolution operation using each of the plurality of kernels K1 is performed on the input layer matrix, and the result of the operation is input to a plurality of feature maps of the first convolutional layer CL1. Here, each of the plurality of kernels K1 may use a matrix having a predetermined size whose elements of the matrix are weights w. In addition, each of the plurality of feature maps of the first convolutional layer CL1 is a matrix having a predetermined size.

Next, a pooling operation using a plurality of kernels K2 is performed on the plurality of feature maps of the first convolutional layer CL1. The plurality of kernels K2 are also matrices of a predetermined size each of which consists of weights w. The operation result of this subsampling is input to a plurality of feature maps of the first pooling layer PL1. The plurality of feature maps of the first pooling layer PL1 are each a matrix of a predetermined size.

Subsequently, a convolution operation is performed on the plurality of feature maps of the first polling layer PL1 using the kernel K3, which is a matrix having a predetermined size including weights w, for each element of the matrix. A second convolutional layer CL2 consisting of a map is configured. Next, a second pooling consisting of a plurality of feature maps is performed by performing a subsampling operation using the kernel K4, which is a matrix of a plurality of weights w, on the plurality of feature maps of the second convolutional layer CL2. Configure the layer PL2. Each second pooling layer PL2 is also a matrix of a predetermined size.

Then, a convolution operation using a plurality of kernels K5 is performed on the plurality of feature maps of the second polling layer PL2. The plurality of kernels K5 are also matrices of a predetermined size whose elements are the weights w. A complete connection layer FL is generated according to a result of a convolution operation using a plurality of kernels K5. In other words, a result of a convolution operation using the plurality of kernels K5 is input to the plurality of nodes f1 to fn.

Each of the nodes f1 to fn of the fully connected layer FL performs a predetermined operation using a transfer function or the like on an input from the second polling layer PL2, and applies a weight w to the operation. Input to each node of middle layer (ML). Accordingly, the plurality of nodes m1 to mm of the middle layer ML performs a predetermined operation on the value input from the fully connected layer FL, and outputs the resultant intermediate value. As described above, the output value of each of the plurality of output nodes m1 to mm is a probability value corresponding to each of the plurality of control signals.

Referring again to FIG. 5, each of the left neural network 211 and the right neural network 213 includes a plurality of layers IL, CL, PL, FL, and ML, as shown in FIG. 6, and the left neural network 211. Receives the left image (L) and outputs an intermediate value through a plurality of operations of a plurality of layers (IL, CL, PL, FL, ML), and the right neural network 213 receives the right image (R) The intermediate value is output through a plurality of operations of the hierarchical layers IL, CL, PL, FL, and ML.

At this time, the intermediate value output from each of the left neural network 211 and the right neural network 213 is input to the merging network 215. The merged network 215 includes an output layer OL consisting of a plurality of output nodes o1, o2,..., Om. That is, the intermediate value output from each of the left neural network 211 and the right neural network 213 is weighted (wL1, wL2, ... wLm, wR1, wR2, ... wRm) is applied to the output of the merged network 215 It is input to the plurality of output nodes o1, o2, ..., om of the layer OL.

The output layer OL may be composed of a plurality of output nodes (or sigmoids: o1, o2, ..., om). Each of the plurality of output nodes o1, o2, ..., om may correspond to a predetermined control signal. An intermediate value, which is an output of each of the plurality of output nodes o1, o2, ..., om, is a probability value.

For example, the first output node o1 corresponds to a volume up, which is a first control signal for increasing the volume of the audio signal, and an intermediate value, which is an output of the first intermediate node o1, is an input left image. Alternatively, the right image may be an eye region image (for example, an eye raised upward) to perform a volume up which is a first control signal. As another example, the third output node o3 corresponds to a channel change channel, which is a third control signal for changing a channel of the audio signal for noise removal, and an intermediate value that is an output of the third output node o3. Denotes the probability that the input left image or the right image is an eye region image (eg, frowning) requiring a change channel, which is a third control signal.

As described above, each of the plurality of layers of the artificial neural network 210 is composed of a plurality of operations, and the calculation result of any one of the layers is applied to the next layer by applying a weight w.

For example, a convolution operation is performed on a matrix of the input layer IL by using a plurality of kernels K1, each of which is a matrix having a weight w, and thus, a first convolutional layer CL1 composed of a plurality of feature maps. ). Here, it is assumed that the number of kernels K1 is six and the size of six kernels K1 is 256 ㅧ 256. Then, six weights w of 256 ㅧ 256 존재 exist, and since the number of input matrices is one, calculation of 1 ㅧ 256 ㅧ 256 ㅧ 6 is performed.

As another example, each of the nodes f1 to fn of the fully connected layer FL performs a predetermined operation using a transfer function or the like on an input from the second polling layer PL2, and the result of the calculation is a weight (w). ) Is applied to each of the plurality of nodes m1 to mm of the middle layer ML. Here, it is assumed that the complete connection layer FL is composed of 64 nodes f1 to f64 (n = 64), and the middle layer ML is composed of 32 nodes m1 to mm (m = 32). In this case, the weights w connecting the nodes f1 to f64 and the nodes m1 to mm are 64 ㅧ 32, and 64 ㅧ 32 operations are performed.

Meanwhile, referring to FIG. 7, a plurality of kernels of at least one convolution layer among the plurality of convolution layers according to an embodiment of the present invention extract a feature of a shape from an eye region image. As shown in FIG. 7, the first to fourth kernels perform a convolution operation on an image (pixel value) of an eye region image to (a) the eyebrow, (b) the upper part of the eye, and (c) the lower part of the eye. And (D) can be extracted to distinguish the shape of the pupil from other shapes.

In addition, referring to FIG. 8, a plurality of kernels of at least one convolution layer among the plurality of convolution layers according to an embodiment of the present invention extract a feature of a location from an eye region image. As shown in FIG. 8, the fifth to eighth kernels perform a convolution operation on a coordinate mask (coordinate value) of an eye region image, so that the eyebrows, the upper part of the eye, and the eye of the eye are The characteristics of the position of the lower part of the eye and (h) the eye can be extracted separately from the others. As described above, the artificial neural network 210 according to the embodiment of the present invention distinguishes and analyzes the features of the extracted shape and the features of the location to more accurately determine the control signal desired by the user.

On the other hand, when the image as the training data is input to the artificial neural network 210, the artificial neural network 210 will output the calculation result through a plurality of operations to which the weight (w) as described above.

The output value of each of the plurality of output nodes o1, o2, ..., om of the output layer OL is a probability that the eye region image, which is input training data, is an image corresponding to a control signal corresponding to the corresponding node. Indicates. Since the training data is a target value, that is, a corresponding control signal is known data, when the learning module 220 inputs the eye region image, which is training data, to the artificial neural network 210, the output of the artificial neural network 210 is the target. The weight w of the artificial neural network 210 should be corrected to be a value. Since the output value is a probability value, the learning module 220 must correct the weight w of the artificial neural network 210 so that the output value corresponding to the target value among the output values of the output nodes o1 to om has the highest probability value. However, the artificial neural network 210 that is not sufficiently learned has a difference between an output value and a target value. Therefore, whenever the learning module 220 inputs the eye region image which is the training data, the learning module 220 calculates the weight w of the artificial neural network 210 through a back-propagation algorithm such that the difference between the target value and the output value is minimized. Do the learning to correct. As described above, in the deep learning according to an embodiment of the present invention, a weight value is applied to the calculation of the artificial neural network 210 so that a target value corresponding to the input learning data is determined and the difference between the output value and the target value is minimized. Modify (w).

The learning module 220 repeatedly performs the deep learning procedure as described above using a plurality of different learning data until it is determined that the artificial neural network 210 has been sufficiently learned. Here, the learning module 220 determines that the artificial neural network 210 is sufficiently learned if the difference between the target value and the output value is less than or equal to a predetermined value and the output value does not change even when some learning data (eye region image) is input. Can be.

Next, a method for controlling the hands-free eyeglass type hearing aid device according to the embodiment of the present invention will be described. 9 is a flowchart illustrating a method for controlling the hands-free eyeglass type hearing aid device according to the embodiment of the present invention.

The controller 170 generates an eye region image by capturing an eye region of the user through the camera unit 110 in operation S110. The eye region image includes an image composed of pixel values corresponding to each pixel and a coordinate mask composed of three-dimensional coordinate values corresponding to each pixel of the image.

The controller 170 analyzes the eye region image in step S120 to identify a control signal corresponding to the eye region image. For example, the control unit 170 is an eye region image, the user intentionally increases the volume of the audio signal output from the audio output module 133, the expression of the eye region (for example, the eyes raised up, etc.) to lower the volume Is detected, the control signal to increase or decrease the volume is detected. In addition, the controller 170 may change the band or channel to remove the noise when the eye region image is analyzed to be an expression of the eye region (eg, frowning) that recognizes the moment when the noise of the audio signal is outside the acceptable limit. To detect a control signal. If the eye region image is an expression of the eye region in which the eye gazes in the direction in which the sound is generated (for example, the user observes a specific direction), the controller 170 sets the direction of the directional microphone MIC in the same direction as the eye. The control signal to be changed is detected.

Next, the control unit 170 controls the hearing aid device 100 according to the control signal previously detected in step S130. That is, the controller 170 may increase or decrease the volume of the audio signal output to the audio output module 133 of the audio unit 130 according to the detected control signal. In addition, the controller 170 may change a band or channel of the audio signal output to the audio output module 133 of the audio unit 130 to remove noise according to the detected control signal. The controller 170 may change the directing direction of the directional microphone MIC, which is the audio input module 131 of the audio unit 130, in the same direction as the line of sight according to the detected control signal.

Meanwhile, according to the present invention, the artificial neural network 210 may be used to detect the control signal through the analysis of the eye region image of the controller 170 described above. For this analysis, learning about the artificial neural network 210 should be made. Then, the learning method of the artificial neural network according to an embodiment of the present invention will be described. 10 is a flowchart illustrating a learning method of an artificial neural network according to an embodiment of the present invention.

Referring to FIG. 10, in operation S210, the learning module 220 inputs an eye region image including a left image L and a right image R, which are learning data, into the artificial neural network 210.

Then, the artificial neural network 210 calculates an output value through a plurality of operations to which weights w of the plurality of layers are applied in step S220.

Then, the learning module 220 uses the target value which is a control signal corresponding to the eye region image input in step S230, so that the output value of the artificial neural network 210 has a minimum difference from the target value. The weight of the artificial neural network 210 is corrected through a -propagation algorithm.

For example, if the learning data is an eye region image including an expression of an eye region for increasing or decreasing the volume of an audio signal, a corresponding target value may be a control signal for increasing volume or a control signal for decreasing volume. have. In addition, if the learning data is an eye region image including an expression of an eye region recognizing noise of an audio signal, a target value corresponding thereto may be a control signal for changing a band or a channel of the audio signal. If the learning data is an eye region image including an expression of an eye region in which a gaze moves in a direction in which a sound is generated, the target value is a control signal for controlling the direction of the directional microphone MIC to direct the sound direction. Can be.

This learning procedure is repeatedly performed through a plurality of different learning data, that is, eye region images. The learning module 220 may determine that the artificial neural network 210 has been sufficiently learned if the difference between the target value and the output value is less than or equal to a predetermined value and the output value does not change even when any learning data is input.

Next, a learning method of an artificial neural network according to another embodiment of the present invention will be described. 11 is a flowchart illustrating a learning method of an artificial neural network according to another embodiment of the present invention.

Referring to FIG. 11, in step S310, the learning module 220 trains the left neural network 211 using the left image L of the eye region image. When the learning module 220 inputs the left image L to the left neural network 211 in the step S310, the left neural network 211 receives a median value through a plurality of operations to which weights w of a plurality of layers are applied. Will calculate. Then, the learning module 220 uses the target value, which is a control signal corresponding to the left image L, previously input, so that the intermediate value, which is the output of the left neural network 211, is minimized from the target value. The weight of the left neural network 211 is corrected through the back-propagation algorithm. The step S310 is repeatedly performed using different left images L, but is repeated until the difference between the target value and the median value is less than a predetermined value and there is no change in the output value more than a predetermined number of times.

As described above, like the output value of the entire artificial neural network 210, the intermediate value, which is the output of the left neural network 211, has a corresponding control signal, and the output of the intermediate value corresponds to a control signal corresponding to the input left image. The probability to represent. Accordingly, the learning module 220 may separately train the left neural network 211 from the right neural network 213 and the merged network 215 using the left image L. FIG.

In addition, the learning module 220 learns the right neural network 213 using the right image R of the eye region image in step S320. When the learning module 220 inputs the right image R to the right neural network 213 in step S320, the right neural network 213 receives an intermediate value through a plurality of operations to which weights w of a plurality of layers are applied. Will calculate. Then, the learning module 220 uses the target value, which is a control signal corresponding to the right image R, previously input, so that the intermediate value, which is the output of the right neural network 213, is minimized from the target value. The weight of the right neural network 213 is corrected through the back-propagation algorithm.

This step S320 is repeatedly performed using different right images R, but is repeated until the difference between the target value and the median value is less than a predetermined value and the output value is not changed more than a predetermined number of times.

As described above, the learning module 220 may separately learn the right neural network 213 from the left neural network 211 and the merged network 215 by using the right image R, similarly to the left neural network 211. have.

As described above, when the individual learning procedure of the left neural network 211 and the right neural network 213 is completed, the learning module 220 includes a left image (L) and a right image (R), which is the training data in step S330. The entire neural network 210 is trained using all the eye region images. That is, when the learning module 220 inputs an eye region image to the artificial neural network 210, the artificial neural network 210 will calculate an output value through a plurality of operations to which weights w of a plurality of layers are applied. Then, the learning module 220 uses a target value that is a control signal corresponding to the previously input eye region image so that the output value of the neural network 210 has a minimum difference from the target value (Back-propagation algorithm). Correct the weight of the artificial neural network (210) through.

This step S330 is repeatedly performed using different eye region images, but is repeated until the difference between the target value and the output value is less than a predetermined value and the output value is not changed more than a predetermined number of times.

As described above, after the left neural network 211 and the right neural network 213 are individually trained, and the entire artificial neural network 210 is trained, the artificial neural network 210 may have any of the left image L and the right image R. Even if one has an error for various reasons, the control signal can be discriminated without error.

Next, a method for controlling the hands-free eyeglass type hearing aid device according to the embodiment of the present invention will be described. 12 is a flowchart illustrating a method for controlling the hands-free eyeglass type hearing aid device according to the embodiment of the present invention.

The controller 170 generates an eye region image including the left image L and the right image R by capturing the eye region of the user through the camera unit 110 in operation S410. The eye region image includes an image composed of pixel values corresponding to each pixel and a coordinate mask composed of three-dimensional coordinate values corresponding to each pixel of the image.

The control module 230 of the controller 170 inputs an eye region image including the left image L and the right image R to the artificial neural network 210 in step S420.

Then, the control module 230 may calculate an intermediate value of each of the left neural network 211 and the right neural network 213 through the left neural network 211 and the right neural network 213 in step S430.

In addition, the control module 230 may derive an output value by merging intermediate values of the left neural network 211 and the right neural network 213 through the merging network 215 in step S440.

Then, the control module 230 may control the hearing aid device 100 according to the control signal corresponding to the output value derived in step S450. That is, the control module 230 controls the volume of the audio signal output from the audio output module 133 according to the control signal corresponding to the derived output value, or the band and channel of the audio signal output to the audio output module 133. Alternatively, the direction of the directional microphone MIC, which is the audio input module 131, may be controlled.

Next, a method of controlling the hands-free eyeglass type hearing aid device according to another embodiment of the present invention will be described. 13 is a block diagram illustrating a detailed configuration of a controller according to another embodiment of the present invention. 14 is a diagram for describing a feature point in an eye region image, according to another exemplary embodiment.

Referring to FIG. 13, according to another embodiment of the present invention, the controller 170 includes a feature point derivation module 310, a displacement measuring module 320, and a control generation module 330.

The feature point derivation module 310 detects a feature point of a predetermined area of the eye and the eyebrow from the eye region image, which is an image of the area including the eye and the eyebrow photographed by the camera unit 110. Referring to FIG. 14, the feature points in each of the left image R and the right image L include left and right ends, top and bottom ends, and left and right ends, top and bottom ends, and pupils of the eyebrows.

The displacement measuring module 320 is for measuring the displacement of the above-described feature point. In other words, the left and right ends, the top and bottom of the eyebrows, and the left and right ends, the top and bottom of the eye, and the pupil measure the change in any position. The displacement measuring module 320 may measure a change in position through the coordinate masks LM and RM formed of coordinate values (eg, x, y, and z values) of the pixels detected as the feature points.

The control generation module 330 generates a control signal according to the measured displacement of the feature point. For example, the control generation module 330 generates a control signal to increase or decrease the volume of the audio signal output according to the measured displacement of the feature point, or to generate a control signal to change the band or channel to remove noise, A control signal for changing the directing direction of the directional microphone MIC in the same direction as the line of sight may be generated.

Next, a method of controlling the hands-free eyeglass type hearing aid device according to another embodiment of the present invention will be described. 15 is a flowchart illustrating a detailed configuration of a control unit according to another embodiment of the present invention.

Referring to FIG. 15, the feature point derivation module 310 of the controller 170 continuously receives an eye region image, which is an image of an area including an eye and an eyebrow generated by the camera unit 110 in operation S510. Here, the eye region image includes an image (RI, LI) consisting of pixel values corresponding to each pixel and a coordinate mask (RM, LM) consisting of three-dimensional coordinate values corresponding to each pixel of the image.

The feature point derivation module 310 detects a feature point of a predetermined region of the eye and the eyebrow in the eye region image in step S520. Referring to FIG. 14, the feature points in each of the left image R and the right image L include left and right ends, top and bottom ends, and left and right ends, top and bottom ends, and pupils of the eyebrows. Here, the feature point derivation module 310 detects the contours of the eye and the eyebrows according to the morphology technique, and according to the coordinate masks LM and RM among the pixels corresponding to the contours, Both left and right ends, top and bottom, and pupils of the eye can be detected as feature points.

Displacement measurement module 320 is for measuring the displacement of the above-described feature point in step S520. In other words, the left and right ends, the top and bottom of the eyebrows, and the left and right ends, the top and bottom of the eye, and the pupil measure the change in any position. The displacement measuring module 320 may measure a change in position through the coordinate masks LM and RM formed of coordinate values (eg, x, y, and z values) of the pixels detected as the feature points.

The control generation module 330 generates a control signal according to the displacement of the feature point measured in step S540. For example, the control generation module 330 may detect eye blinks, eye changes, and eye distortions through the displacement of the feature points. Accordingly, the control generation module 330 generates a control signal for increasing or decreasing the volume of the audio signal, generating a control signal for changing the band or channel to remove noise, or directing the direction of the directional microphone (MIC). It can generate a control signal to change the same direction as the eye.

Next, the control generation module 330 controls the spectacle-type hearing aid device 100 through the control signal previously generated in step S550. As described above, according to the present invention, the shape and position of the eye and the eyebrow of the eye region image are analyzed through feature points to generate a control signal that the user intends to consciously manipulate through the eye, such as volume control, and to hear without hand manipulation. You can operate the device. Furthermore, according to the present invention, the eyes and the eyebrows of the eye region image by analyzing the shape and position of the user, although unconscious or unintentional, makes an uncomfortable facial expression due to noise, or recognize the eye that the eyes are in the direction of sound The control device may generate a control signal to cancel noise or control the hearing aid device to listen to the sound in a direction in which the gaze is possible.

On the other hand, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable through various computer means can be recorded on a computer-readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in computer software. For example, the recording medium may be magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs, DVDs, or magnetic-optical media such as floptical disks. magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level languages that can be executed by a computer using an interpreter as well as machine language such as produced by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been described using some preferred embodiments, these embodiments are illustrative and not restrictive. As such, those of ordinary skill in the art will appreciate that various changes and modifications can be made according to equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: eyeglass frame 100: hearing aid
110: camera unit 130: audio unit
140: input unit 150: projection unit
160: storage unit 170: control unit
210: artificial neural network 211: left neural network
213: right neural network 215: merged network
IL: input layer CL: convolutional layer
PL: Pooling Layer FL: Fully Connected Layer
ML: Middle OL: Output Layer
220: learning module 230: control module
310: feature point derivation module 320: displacement measurement module
330: control generation module

Claims (23)

In the hands-free eyeglass type hearing aid device,
Eyeglass frames including rims, bridges, temples and tips;
An audio unit installed at the tip part to output an audio signal; And
A camera unit installed in at least one of the rim, the bridge, and the temple to capture an eye region image, which is an image of an area including eyes and eyebrows of the left and right sides of a user; And
And a controller configured to control the hands-free eyeglass type hearing aid according to the shape and position of the eye and eyebrow of the analyzed eye region image by analyzing the image photographed by the camera unit. The method of claim 1,
The control unit
According to the shape and position of the eye and eyebrow of the analyzed eye region image
Hands-free eyeglass type hearing aid device, characterized in that for changing the volume, band and channel of the audio signal output to the audio unit. The method of claim 1,
The audio unit
And a plurality of audio receiving modules installed in predetermined regions of the rim, the bridge, and the temple to receive audio signals.
The control unit
Hands-free glasses-type hearing aid, characterized in that for controlling the direction of the directional microphone according to the shape and position of the eye and eyebrow of the analyzed eye region image. The method of claim 1,
The control unit
A feature point detection module for detecting feature points of a predetermined region of the eye and the eyebrow in an eye region image which is an image of the area including the eye and the eyebrow;
Displacement measurement module for measuring the displacement of the feature point; And
And a control generation module for generating and outputting a control signal for controlling the hands-free glasses-type hearing aid according to the measured displacement of the feature point. The method of claim 4, wherein
The feature point is
Left and right ends of the eyebrows, the top and bottom of the eyebrows,
Left and right ends of the eyes, the top and bottom of the eyes and
Hands-free eyeglass type hearing aid comprising a pupil. In the hands-free eyeglass type hearing aid device,
Eyeglass frames including rims, bridges, temples and tips;
An audio unit installed at the tip part to output an audio signal;
It is installed in at least one area of the rim, bridge and temple to photograph the area including the eyes and eyebrows of the left and right sides of the user to obtain an image composed of pixel values of a plurality of pixels and a three-dimensional coordinate value of the plurality of pixels. A camera unit generating an eye region image including a coordinate mask including the coordinate mask; And
And a controller configured to control the spectacle hearing aid according to the shape and position of the eye and the eyebrows of the analyzed eye region image by analyzing the eye region image. The method of claim 6,
The control unit
The output of any one layer includes a plurality of layers constituting the next layer through a plurality of operations to which weights for determining the strength of the inter-layer connection are applied, and when the eye region image is input, the plurality of operations are performed. An artificial neural network for outputting a result of the operation;
A learning module for learning the artificial neural network using the eye region image and a control signal which is a target value; And
And a control module for inputting the eye region image to the artificial neural network to derive an output value and to control the spectacle hearing aid using a control signal corresponding to the derived output value when the learning of the artificial neural network is completed. Hands-free eyeglass type hearing aid. The method of claim 7, wherein
The artificial neural network
A left neural network for outputting an intermediate value through an operation in which weights of a plurality of layers are applied to a left image, which is an image of an area including an eye and an eyebrow of a left side of the user;
A right neural network for outputting an intermediate value through an operation in which weights of a plurality of layers are applied to a right image, which is an image of an area including the eyes and eyebrows of the user's right side; And
And a merged network configured to calculate an output value of the artificial neural network through a calculation in which weights are applied to the intermediate values of the left and right neural networks, respectively. The method of claim 8,
Each of the left and right neural networks
A plurality of convolutional layers,
A plurality of kernels of at least one convolution layer of the plurality of convolution layers
Have the same size as the input layer,
Each convolution operation using a plurality of kernels of the at least one convolution layer
Hands-free spectacles-type hearing aid, characterized in that the extraction features of the shape of the eyebrows, the upper part of the eye, the lower part of the eye and the pupil. The method of claim 8,
Each of the left and right neural networks
A plurality of convolutional layers,
Each of the convolution operations using a plurality of kernels of at least one convolution layer of the plurality of convolution layers
Hands-free spectacles-type hearing aid, characterized in that the extraction of the features of the eyebrow, the upper part of the eye, the lower part of the eye and the pupil. The method of claim 6,
The learning module
Inputting an eye region image having a predetermined control signal as the target value to the artificial neural network,
Hands-free glasses type hearing aid, characterized in that for correcting the weight so that the output value of the artificial neural network is the minimum difference from the target value. The method of claim 6,
The learning module
Input the left image of the eye region image to the left neural network, and correct the weight of the left neural network through a despreading algorithm so that the intermediate value of the left neural network is minimized from the target value. Learning,
The right neural network is inputted to the right neural network, and the weight of the right neural network is corrected by a despreading algorithm so that the intermediate value of the right neural network is minimized from the target value. Learning,
The eye region image including the left image and the right image is input to the artificial neural network, and the weight of the artificial neural network is corrected through a despreading algorithm such that an output value of the artificial neural network has a minimum difference from a target value. Hands-free glasses-type hearing aid, characterized in that for learning the artificial neural network. The method of claim 6,
The control module
Inputting an eye region image including a left image, which is an image of a region including an eye and an eyebrow, on the left side of the user, and a right image, which is an image of an area including an eye and an eyebrow, on the right of the user, to the artificial neural network;
Derive an output value of the artificial neural network for the eye region image,
Determine a control signal corresponding to the output value,
Hands-free glasses-type hearing aid, characterized in that for controlling the volume of the audio signal output to the audio unit, changing the band and channel of the audio signal output to the audio unit, or controls the direction of the directional microphone according to the determined control signal Device. A method for controlling a hands free spectacle hearing aid,
The camera unit is installed on at least one of the rim, the bridge, and the temple of the spectacle frame to capture an area including the eyes and eyebrows of the left and right sides of the user.
Generating an eye region image which is an image of an area including eyes and eyebrows of the left and right sides of a user;
And controlling the hands-free eyeglass type hearing aid device according to the shape and position of the eye and eyebrow of the analyzed eye region image by analyzing the image photographed by the camera unit. Way. The method of claim 14,
The step of controlling the hands-free eyeglass type hearing aid device
Detecting a feature point of a predetermined area of the eye and the eyebrow in an eye area image, which is an image of the area including the eye and the eyebrow;
Measuring the displacement of the feature point; And
And changing the volume, band, and channel of the output audio signal according to the measured displacement of the feature point, or controlling the directing direction of the plurality of directional microphones receiving the audio signal. Method for controlling. The method of claim 15,
The feature point is
Left and right ends of the eyebrows, the top and bottom of the eyebrows,
Left and right ends of the eyes, the top and bottom of the eyes and
Characterized by including the pupil
A method for controlling a hands free spectacle hearing aid. A method for controlling a hands free spectacle hearing aid,
An image consisting of pixel values of a plurality of pixels and three-dimensional coordinates of the plurality of pixels by photographing an area including the eyes and eyebrows of the left and right sides of the user through a camera unit installed in at least one of the rim, the bridge, and the temple. Generating an eye region image including a coordinate mask including a value;
The control signal corresponding to the eye region image according to the shape and position of the eye and the eyebrow of the eye region image by analyzing the eye region image using an artificial neural network including a plurality of layers of a plurality of calculations to which weights are applied. Detecting; And
And controlling the spectacle hearing aid according to the detected control signal. The method of claim 17,
Detecting the control signal
An intermediate value is calculated through a calculation in which weights of a plurality of layers are applied to a left image, which is an image of an area including the eyes and eyebrows of the left side of the user, and an image of the area including the eyes and eyebrows of the user's right side. Calculating an intermediate value through an operation to which weights of a plurality of layers are applied to the right image;
Calculating an output value of the artificial neural network through a calculation in which a weight is applied to an intermediate value of each of the left neural network and the right neural network; And
And detecting a control signal corresponding to the output value. The method of claim 18,
The calculating of the median value
Through convolution operation using multiple kernels
Extracting features of each of the shape of the eyebrow, the top of the eye, the bottom of the eye, and the pupil. The method of claim 18,
The calculating of the median value
Through convolution operation using multiple kernels
Extracting features of the eyebrow, the top of the eye, the bottom of the eye, and the location of each of the pupils. The method of claim 17,
Before the step of detecting the control signal,
The left image of the eye region image is input to the left neural network of the artificial neural network, and the weight of the left neural network is corrected through a despreading algorithm so that the intermediate value of the left neural network is minimized from the target value. Learning the left neural network;
The right image of the eye region image is input to the right neural network of the artificial neural network, and the weight of the right neural network is corrected by a despreading algorithm so that the intermediate value of the right neural network is minimized from the target value. Learning the right neural network; And
The eye region image including the left image and the right image is input to the artificial neural network, and the weight of the artificial neural network is corrected through a despreading algorithm such that an output value of the artificial neural network has a minimum difference from a target value. Learning the artificial neural network by a; a method for controlling a hands-free spectacles-type hearing aid comprising a. The method of claim 17,
The controlling of the eyeglass type hearing aid device is
According to the detected control signal
Control the volume of the audio signal output from the audio output module installed on the tip of the eyeglass frame,
Change a band and a channel of an audio signal output to the audio output module;
And controlling the directing direction of the directional microphone provided in a predetermined area of the rim, bridge, and temple of the spectacle frame. A computer-readable recording medium having recorded thereon a program for performing a method for controlling a hands-free eyeglass type hearing aid,
An image consisting of pixel values of a plurality of pixels and an image composed of pixel values of a plurality of pixels by photographing an area including eyes and eyebrows of the left and right sides of a user through a camera unit installed in at least one of the rim, the bridge, and the temple of the spectacle frame Generating an eye region image including a coordinate mask including a dimensional coordinate value;
Analyzing the eye region image and detecting a control signal corresponding to the eye region image according to the shape and position of the eye and eyebrow of the eye region image; And
And controlling the spectacle hearing aid according to the detected control signal. A computer readable recording medium having recorded thereon a program for performing a method for controlling a hands-free spectacle hearing aid.

Images