KR20170111450A - Hearing aid apparatus, portable apparatus and controlling method thereof - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a hearing aid, a portable device, and a control method thereof, and a portable device includes a sound input unit including a plurality of microphones for receiving a sound signal and a hearing aid provided with a video input unit for receiving image data, A communication unit for receiving data; A storage unit for storing a voice database; Determines whether or not a speaker is present at the ignition position determined using the received image data, and if it is determined that a speaker is present at the ignition position, uses the voice database from the received voice signal And at least one processor for adjusting the identified voice signal so that the level of the hearing loss frequency region is increased with respect to the voice signal corresponding to the identified speaker and controlling the adjusted voice signal to be transmitted to the hearing aid through the communication unit, The adjusted voice signal can be output by the hearing aid. Thereby, by selectively amplifying and outputting the speech signal of the speaker existing at the speech position of the speech signal, the hearing impaired person can listen to the speech of the conversation partner without inconvenience, thereby increasing the concentration of conversation.

Description

[0001] HEARING AID APPARATUS, PORTABLE APPARATUS AND CONTROLLING METHOD THEREOF [0002]

The present invention relates to a hearing aid, a portable device, and a control method thereof, and more particularly to a hearing aid, a portable device, and a control method thereof capable of amplifying a voice signal of a speaker in conversation.

Various types of electronic products are being developed and distributed by the development of electronic technology. For example, the use of a variety of mobile portable devices such as smart phones, smart pads (tablets), PDAs and the like is increasing and these portable devices can be provided as body wearable devices.

As an example of a body wearing device, a hearing aid is a device for amplifying sound attached to a patient diagnosed with hearing loss by a hearing test, and it is generally operated in accordance with hearing power of a hearing impaired person.

Therefore, when a sound is simultaneously input from various sound sources including a plurality of speakers, the hearing loss frequency region is amplified with respect to all input signals, and it may be difficult to hear and understand the sound that requires hearing loss.

Accordingly, it is necessary to provide a means for helping the hearing impaired person to concentrate on the conversation partner without confusion even in an environment where various sounds are generated.

A portable device according to an embodiment of the present invention includes a communication unit for receiving voice signals and image data from a hearing device provided with a voice input unit including a plurality of microphones for receiving voice signals and a video input unit for receiving video data; A storage unit for storing a voice database; Determines whether or not a speaker is present at the ignition position determined using the received image data, and if it is determined that a speaker is present at the ignition position, uses the voice database from the received voice signal And at least one processor for adjusting the identified voice signal so that the level of the hearing loss frequency region is increased with respect to the voice signal corresponding to the identified speaker and controlling the adjusted voice signal to be transmitted to the hearing aid through the communication unit, The speech signal adjusted to increase the level of the hearing loss frequency region can be output by the hearing aid. Thus, the speaker present at the utterance position of the voice signal is regarded as the opponent of the conversation, and the voice signal of the opponent is selectively amplified and output, so that the wearer can easily listen to the voice of the opponent without inconvenience.

The hearing aid is a body wearing device, and the image input unit includes a front camera installed on the front face of the hearing aid. The processor uses the image data photographed on the front face of the wearer with the front camera, Can be determined. Thus, it is possible to determine whether or not the user is conversing from the video data, and utilize it to specify the conversation partner.

The image input unit further includes a rear face camera provided at a position through which at least one line of sight of the wearer of the hearing aid passes and acquires eye information of the wearer. The processor uses the obtained eye information to determine whether or not the wearer stared at the speaker You can decide more. Thus, it is possible to confirm whether or not the user is gazing at the other party, and it is possible to specify the communication party more accurately.

The hearing aid is further provided with a direction sensor and the processor can further use the received sensing value to control the communication part to receive the sensing value of the direction sensor from the hearing aid and to determine whether a speaker is present at the determined speech position . This improves the accuracy of the position of the other party.

The processor can estimate the speech position of the speech signal using at least one of a time difference in which the speech signal reaches the plurality of microphones, a distance between the plurality of microphones, and a number of the microphones. Accordingly, various algorithms for estimating the speech position of the speech signal can be utilized to specify the conversation partner.

The speech database stores information on speech signals corresponding to a plurality of speakers and a plurality of speakers. The processor identifies and separates the speaker's speech signal from the received speech signal based on the speech database, The level of the hearing loss frequency region with respect to the signal can be increased. Thus, the voice required for the conversation can be selectively adjusted, which can help improve the conversation concentration of the hearing impaired person.

The processor may determine whether information corresponding to the received voice signal is present in the voice database and store information about the voice signal in the database if it is determined that the information does not exist. Accordingly, the information about the perimeters of the hearing impaired person is constantly updated, so that it can be utilized not only for future speaker recognition but also can build a user's own speech database.

The apparatus further includes a user input section for receiving user input, and the processor can store the information received through the user input section in the voice database in correspondence with the voice signal. Accordingly, it is possible to provide various information that allows the user to easily identify the communication partner.

According to another aspect of the present invention, there is provided a hearing aid comprising: a voice input unit including a plurality of microphones for receiving voice signals; A video input unit for receiving video data; A communication unit for transmitting the received voice signal and video data to a portable device; An audio output unit for outputting a voice signal; And at least one processor for controlling the communication unit to receive the voice signal adjusted to increase the level of the hearing loss frequency region with respect to the voice signal corresponding to a predetermined speaker from the portable device and to control the voice output unit to output the adjusted voice signal The image input unit includes a front camera installed on the front of the hearing aid. Using the image data captured by the front camera, it is determined whether or not a speaker is present at a speaking position of a speech signal, and a speaker is present at a speaking position The voice signal corresponding to the identified speaker can be adjusted using the voice database and transmitted from the portable device to the hearing aid. Accordingly, by simply looking at the other party wearing the hearing aid, the other party of the conversation is confirmed, and the voice signal is amplified and output, so that conversation concentration can be improved even in a noisy environment.

The image input unit further includes a rear camera provided at a position through which at least one line of sight of the wearer's wearer passes and acquires eye information of the wearer, and it may be further determined whether the wearer looks at the speaker using the acquired eye information have. Accordingly, since the user utilizes whether or not the user is gazing at the other party, the accuracy of the speaker recognition is further improved.

The display unit may further include a display unit provided at a position where at least one line of sight of the wearer passes, and the processor may control the display unit to display information on the speaker identified using the speech database and the speech database. Accordingly, various information on the communication partner is provided to the user, thereby facilitating the convenience.

According to another aspect of the present invention, there is provided a method for controlling a portable device, the method comprising: receiving a voice signal and image data from a hearing device provided with a voice input unit including a plurality of microphones for receiving voice signals and a video input unit for receiving image data; ; Determining an utterance position of the received voice signal; Determining whether a speaker is present at a determined firing position using the received image data; Adjusting the identified speech signal so that the level of the hearing loss frequency region is increased with respect to the speech signal corresponding to the speaker identified using the speech database from the received speech signal when the speaker is determined to be present at the speech position; And transmitting the adjusted voice signal from the portable device to the hearing aid so as to be outputted by the hearing aid. Thus, the speaker present at the utterance position of the voice signal is regarded as the opponent of the conversation, and the voice signal of the opponent is selectively amplified and output, so that the wearer can easily listen to the voice of the opponent without inconvenience.

The step of determining whether or not a speaker is present includes the step of determining whether or not a speaker is present in the image input unit, To determine whether a speaker is present at the determined firing position. Thus, it is possible to determine whether or not the user is conversing from the video data, and utilize it to specify the conversation partner.

The image input unit further includes a rear camera provided at a position through which at least one line of sight of the wearer's wearer passes and acquiring eye information of the wearer and determining whether the wearer looks at the speaker using the acquired eye information As shown in FIG. Thus, it is possible to confirm whether or not the user is gazing at the other party, and it is possible to specify the communication party more accurately.

The step of determining the firing position may estimate the firing position of the speech signal using at least one of a time difference in which the speech signal reaches the plurality of microphones, a distance between the plurality of microphones, and a number of the microphones. Accordingly, various algorithms for estimating the speech position of the speech signal can be utilized to specify the conversation partner.

Wherein the speech database further stores information on speech signals corresponding to a plurality of speakers and a plurality of speakers and further comprises the step of identifying and separating the speaker's speech signal from the received speech signal based on the speech database, The adjusting step can be adjusted so that the level of the hearing loss frequency region is increased with respect to the voice signal of the separated speaker. Thus, the voice required for the conversation can be selectively adjusted, which can help improve the conversation concentration of the hearing impaired person.

Determining whether information corresponding to the received voice signal is present in the voice database; And if the information is determined not to exist, storing information on the voice signal in the database. Accordingly, the information about the perimeters of the hearing impaired person is constantly updated, so that it can be utilized not only for future speaker recognition but also can build a user's own speech database.

According to another aspect of the present invention, there is provided a method of controlling a hearing aid system including a hearing aid and a portable device, the method comprising: receiving a voice signal from a plurality of microphones; Receiving image data from a front face camera installed on a front face of the hearing aid; Determining an utterance position of the received voice signal; Determining whether a speaker is present at a determined firing position using the received image data; Adjusting the identified speech signal so that the level of the hearing loss frequency region is increased with respect to the speech signal corresponding to the speaker identified using the speech database from the received speech signal when the speaker is determined to be present at the speech position; And outputting the adjusted voice signal by the hearing aid. Accordingly, by simply looking at the other party wearing the hearing aid, the other party of the conversation is confirmed, and the voice signal is amplified and output, so that conversation concentration can be improved even in a noisy environment.

Further comprising the step of acquiring eye information of a wearer from a rear camera which is provided at a position through which at least one eye of the wearer of the hearing aid passes and acquires eye information of the wearer, And determining whether or not to perform the step of determining whether or not to perform the process. Accordingly, since the user utilizes whether or not the user is gazing at the other party, the accuracy of the speaker recognition is further improved.

The hearing aid may further include a display unit provided at a position where at least one line of sight of the wearer passes through, and displaying information on the speaker identified using the speech database and the speech database by the display unit have. Accordingly, various information on the communication partner is provided to the user, thereby facilitating the convenience.

1 is a diagram showing an example of a hearing aid system according to an embodiment of the present invention,
2 is a perspective view showing a hearing aid according to an embodiment of the present invention,
3 is a block diagram showing a configuration of a hearing aid according to an embodiment of the present invention,
4 is a block diagram showing the configuration of a portable device according to an embodiment of the present invention,
5 is a perspective view illustrating a portable device implemented with a smart watch according to an embodiment of the present invention,
6 and 7 are diagrams for explaining a method of estimating a speaker's position in a portable device according to an embodiment of the present invention,
8 is a view for explaining a method of using eye information in a portable device according to an embodiment of the present invention,
9 is a flowchart illustrating a method of controlling a hearing aid system according to an embodiment of the present invention,
FIGS. 10 to 12 are views for explaining a process in which the hearing impaired person communicates with the other party using the hearing device and the portable device according to the control method shown in FIG. 9. FIG.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art to which the present invention pertains. The present invention is not limited to the embodiments described herein, but may be embodied in various different forms.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. Also, in certain cases, there may be a term selected arbitrarily by the applicant, in which case the meaning thereof will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term, not on the name of a simple term, but on the entire contents of the present invention.

Also, in the examples, terms such as " comprising " or " having " are used to specify that a feature, a number, a step, an operation, an element, , Steps, operations, components, or combinations thereof, are not intended to be < / RTI >

As used herein, the term "part " refers to a hardware component, such as a software, FPGA, or ASIC, and the" part " However, "part" is not meant to be limited to software or hardware. "Part" may be configured to reside on an addressable storage medium and may be configured to play back one or more processors. Thus, by way of example, and not limitation, "part (s) " refers to components such as software components, object oriented software components, class components and task components, and processes, Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and "parts " may be combined into a smaller number of components and" parts " or further separated into additional components and "parts ".

In the following description of the present invention, the same reference numerals will be used throughout the drawings to refer to the same or like parts throughout the specification. .

1 is a diagram showing an example of a hearing aid system 1 according to an embodiment of the present invention.

1, a hearing aid system 1 according to an embodiment of the present invention includes a hearing aid apparatus 10 and a portable apparatus 20 provided to be capable of communicating with the hearing aid apparatus 10, .

The hearing aid apparatus 10 receives an analog sound (for example, a voice signal) from the outside and outputs the amplified sound to the wearer. Here, the amplified sound may correspond to a speech signal in which the level of the hearing loss is increased, that is, the level of the hearing loss frequency region of the hearing aid wearer.

In one embodiment of the present invention, the hearing aid 10 is a body wearable type (i.e., wearable) device including a microphone provided as a sound input portion for receiving sound and an earphone provided as a sound output portion for outputting amplified sound device), for example, a head wearable type which is worn on the head of the hearing impaired person, a type of eyeglasses as shown in Fig. 1, and the like.

In one embodiment, the hearing aid 10 may be implemented as a digital hearing aid that converts an analog signal into a digital signal, processes the analog signal, converts the processed digital signal into an analog signal, and outputs the converted analog signal.

In another embodiment, the hearing aid 10 can be implemented to transmit an analog signal to the portable device 20, and to receive and output the adjusted analog signal from the portable device 20. In this case, signal processing such as level adjustment of the hearing loss frequency region can be performed in the portable device 20. [

In one embodiment of the present invention, the portable device 20 may include a smart pad such as a smart phone, a tablet, a personal digital assistant (PDA) Portable information terminals), and the like.

In one embodiment, the portable device 20 is a wearable device (also referred to as a digital accessory, a smart accessory, or an app apparel) that can be worn on the wearer's body, such as a smart watch, a smart band, .

The portable device 20 will be described later in more detail with reference to FIGS. 4 and 5. FIG.

2 is a perspective view illustrating a hearing aid 100 according to an embodiment of the present invention.

As shown in FIG. 2, in one embodiment of the present invention, the hearing aid 100 is provided in a form such as a pair of glasses that the wearer can wear on his or her body.

In FIG. 2, an example of a wearing form is shown as an example of a wearing type, but the present invention is not limited thereto. That is, in the present invention, the hearing aid 100 includes a sound input unit 160 for receiving a sound signal, a sound output unit 170 for outputting an amplified sound signal, and an image input unit 180 for receiving image data on the wearer's front And can be implemented in various forms that can be installed.

2, the voice input unit 160 includes a plurality of microphones 161, 162 and 163 (hereinafter, also referred to as a microphone), and each of the microphones 161, 162 and 163 has a predetermined interval d And is installed in the hearing apparatus 100.

The voice output unit 170 outputs a voice signal whose level of the hearing loss frequency range of the wearer of the hearing aid device 100 is increased and may have the form of earphones 171 and 172 as shown in FIG.

The image input unit 180 includes a front camera 181 for photographing the front face of the wearer of the hearing aid. The image data obtained through the front camera 181 can be used for positioning the speaker.

In one embodiment, the image input unit 180 may further include a rear camera 182 provided at a position through which at least one line of sight of the wearer of the hearing aid device 100 passes and acquires eye information of the wearer. The image data obtained through the backside camera 182 can be used to determine whether or not the wearer is looking at a voice call ringer.

Meanwhile, the hearing device 100 according to an embodiment of the present invention may further include a display unit 140 provided at a position where at least one line of sight of the wearer is talking. As shown in FIG. 2, the display unit 140 may be provided corresponding to the position of the eyes where the rear camera 182 is not installed.

That is, in one embodiment, when the rear camera 182 is provided at a position through which one of the eyes of the wearer passes, the display unit 140 may be provided at a position through which the other eye passes. Here, as another embodiment, the rear car- rier 182 and / or the display unit 140 may be installed at a position where both eyes pass through.

In addition, the hearing aid apparatus 100 may be provided with a communication unit 120 so as to be capable of communicating with the portable device 200. The communication unit 120 includes various communication modules, and the type thereof is not limited.

Hereinafter, the configuration of the hearing aid apparatus 100 will be described in more detail with reference to FIG.

3 is a block diagram showing the configuration of a hearing aid 100 according to an embodiment of the present invention.

3, the hearing aid 100 includes a control unit 110, a communication unit 120, an image processing unit 130, a display unit 140, a user input unit 150, a voice input unit 160, An image input unit 180, a storage unit 190, and a power supply unit 195.

The voice input unit 160 includes a plurality of microphones, a first microphone 161 and a second microphone 162. In one embodiment, the voice input unit 160 may further include a third microphone 163.

The audio output unit 170 includes a first earphone 171 and a second earphone 172. In one embodiment, the audio output unit 170 may include only one of the first earphone 171 and the second earphone 172.

The image input unit 180 includes a front camera 181. In one embodiment, the image input unit 180 may further include a rear camera 182.

In one embodiment, the hearing aid 100 may further include a connector 175 and a direction sensor 165, as shown in Fig.

The control unit 110 controls the overall operation of the hearing aid apparatus 100 and the signal flow between the internal components 120 to 195 of the hearing aid apparatus 100 and performs data processing. The control unit 110 controls power supply from the power supply unit 195 to the internal components 120 to 190. In addition, when the input or the set stored condition of the user is satisfied, the control unit 110 can execute an OS (Operation System) and various applications stored in the storage unit 190. [

In an embodiment of the present invention, the control unit 110 includes a processor 111, a ROM 112, and a RAM 113. [ The processor 111 includes at least one general purpose processor such as a CPU (Central Processing Unit), AP (Application Processor), Microcomputer (MICOM) It is possible to implement various operations of the hearing aid 100 by loading and executing a corresponding program in the RAM 113. [

The processor 111 may include single core, dual core, triple core, quad core, and cores thereof. A processor may include a plurality of processors, for example, a main processor and a sub processor. The sub processor is provided to operate in a standby mode (hereinafter also referred to as a sleep mode) in which only standby power is supplied and does not operate as the hearing aid.

The processor 111, the ROM 112, and the RAM 113 may be interconnected via an internal bus.

In one embodiment, the processor 111 may include a graphics processing unit (GPU) (not shown) for graphics processing. For example, the processor 111 may be implemented in the form of a system on chip (SoC) coupled with a core (not shown) and a GPU (not shown).

Meanwhile, in another embodiment of the present invention, the control unit 110 may include a program for performing a function of detecting an error occurrence in a predetermined function including a main processor, for example, a main function supported by the portable device 200 For example, an integrated circuit (IC) chip provided as a dedicated processor for executing the program.

The control unit 110 includes a communication unit 120, an image processing unit 130, a display unit 140, a user input unit 150, a voice input unit 160, a voice output unit 170, a connector 175, The direction sensor 165, the storage unit 190, and the power supply unit 195, as shown in FIG.

The communication unit 120 includes a communication module capable of communicating with the portable device 200. For example, the communication unit 170 may be a wireless communication unit such as a Bluetooth, a Bluetooth low energy, an IrDA, a Wi-Fi, a Wi-Fi Direct, (Zigbee), UWB (Ultra Wideband), NFC (Near Field Communication), and the like.

In one embodiment, the communication unit 120 may be provided to support human body communication using a magnetic field.

The communication unit 120 transmits the voice signal received through the voice input unit 160 to the portable device 200. [ Here, the audio signal to be transmitted may be an analog signal or a digital signal. When the digitally converted signal is transmitted, the hearing aid 100 may further include an A / D converter.

The communication unit 120 can receive a voice signal from the portable device 200. [ The received voice signal is output to the wearer of the hearing aid 100 through the voice output unit 170. Here, the received voice signal is an analog signal or a signal converted into a digital signal in some cases, as a voice signal whose hearing loss frequency level of the wearer of the hearing aid device 100 is adjusted and the voice signal separated to correspond to the other party of the conversation of the wearer . When the digitally converted signal is received, the hearing aid 100 may further include a D / A converter.

The image processing unit 130 processes the image to be displayed on the display unit 140.

The display unit 140 displays an image processed by the image processing unit 130.

2, the display unit 140 may be provided at a position where at least one line of sight of the wearer passes while wearing the hearing aid device 100, so that the wearer can easily And can be provided for confirmation.

In one embodiment, the display unit 140 may include a touch screen capable of receiving a user's touch input. In addition, the display unit 140 may be implemented as a transparent display.

In one embodiment, the image processing unit 130 displays text, icons, and the like on the display unit 140 to the wearer of the hearing aid apparatus 100 under the control of the control unit 110 to display various information (for example, And a plurality of displayed texts and icons may be stored in the storage unit 190 in advance.

The user input unit 150 receives user input and may include one or more buttons. In one embodiment, the user input unit 150 may include a power button provided at a predetermined position of the hearing aid apparatus 100 to turn on / off the power.

In the hearing aid 100 according to the embodiment, the button may be implemented as a touch button provided on the display unit 140 including a touch screen, not a physical button. In the hearing aid apparatus 100, the button may be displayed as text or an icon on the touch screen.

The sound input unit 160 includes a plurality of microphones 161, 162, and 163 and the control unit 110 controls the microphones 161, 162, and 163 to output sound signals corresponding to the sounds uttered by the speakers Can be collected. Under the control of the controller 110, the microphones 161, 162, and 163 receive voice or sound from the outside and generate electrical signals. The voice signals collected through the respective microphones 161, 162, and 163 are used to estimate the position (direction) of the speaker.

The audio output unit 170 includes earphones 171 and 172 for outputting amplified audio signals. Here, the output audio signal is a signal that increases the level of the hearing loss frequency region of the wearer of the hearing aid. Further, the output audio signal is a noise-removed audio signal.

In one embodiment, a speech signal of a speaker determined as a conversation partner among the speech signals received through the speech input unit 160 may be separated and output through the speech output unit 170. Here, a series of audio signal processing such as determination of the other party of the conversation and separation of the corresponding audio signal can be performed by the control unit 210 of the portable device 200. [

The connector 175 may be used as an interface for connecting the hearing aid 100 to an external device or a power source. In one embodiment, the hearing aid 100 may be connected to the portable device 200 via a connector 175.

The hearing aid apparatus 100 may be powered from a power source via a wired cable connected to the connector 175 or may be charged with a battery provided as a power supply unit 195. [

The image input unit 180 includes a front camera 181 provided on the front of the hearing aid apparatus 100 and capturing a still image or a moving image under the control of the controller 110. Front image data received through the front camera 181 may be transmitted to the portable device 200 through the communication unit 120. [ The controller 210 of the portable device 200 can determine whether or not a speaker is located on the front face of the wearer using the front image data received from the hearing aid device 100, It can be inferred as being in conversation.

In one embodiment, the image input unit 180 may further include a rear camera 182 that photographs at least one eye of the user. The rear image data received through the rear camera 182 can be transmitted to the portable device 200 through the communication unit 120. [ The control unit 210 of the portable device 200 can acquire motion information of the wearer's eye (body) using the backside data received from the hearing aid device 100. [ The acquired eye information is used to determine whether the wearer is looking at the other party, i.e., the speaker, of the conversation.

The front camera 181 and the rear camera 182 include an image sensor for sensing an image transmitted through a lens, and can be implemented as a CCD / CMOS image sensor, for example.

The direction sensor 165 is provided on the front surface of the hearing aid device 100 and can detect presence or absence of a speaker estimated as a conversation partner on the front side. In one embodiment, the direction sensor 165 may further sense motion of the speaker present in the foreground (e.g., movement to the left or right). The sensing value of the direction sensor 165 may be used for determining whether or not a speaker is present using the front image data.

Meanwhile, one embodiment of the hearing aid apparatus 100 may further include a motion sensor (for example, a gyro sensor or an acceleration sensor) for sensing the motion of the hearing aid 100 including the posture change.

The storage unit 190 stores a control program for controlling the hearing aid apparatus 100, an application or an associated graphical user interface provided from a manufacturer or downloaded from the outside, text or images for providing a GUI, Can be stored.

The power supply unit 195 supplies power to the respective components 110 to 190 of the hearing aid 100. The power supply unit 195 includes one or more batteries (not shown) located inside the hearing aid apparatus 100 and supplies operating power to the respective components 110 to 190 under the control of the controller 110 . The power supply unit 195 supplies power to the hearing aid 100 through an external power source (not shown) through a cable (not shown) connected to the connector 175 under the control of the controller 110. [ I.e., to be charged.

Hereinafter, the configuration of the portable device 200 will be described with reference to FIG.

4 is a block diagram showing a configuration of a portable device 200 according to an embodiment of the present invention.

The portable device 200 of the present embodiment can be wirelessly connected to other devices including the hearing aid device 100 using a communication unit 230 described later and can transmit / receive audio signals and video data. In one embodiment, the portable device 200 has a touch screen 241 and is defined as a device capable of transmitting / receiving data through the communication unit 230.

The portable device 200 of the embodiment includes a user input unit 250 including a physical button 251, a keypad displayed on the touch screen 241, and a separate input device (pointing device) And the user input unit 250 may receive a touch by the user's body (e.g., a finger) with respect to the touch screen 241. [ The user input unit 250 transmits various preset control commands or information to the control unit 110 according to a user input including a touch input.

As shown in FIG. 4, the portable device 200 includes a display unit 240 including a touch screen 241, and a touch screen controller 245. The portable device 200 includes a control unit 210, a communication unit 220, an image processing unit 230, a display unit 240, a user input unit 250, a sensing unit 260, an image input unit 280, And a power supply unit 295.

The communication unit 220 includes at least one of a wireless LAN unit 221 and a local communication unit 222. The image processing unit 230 includes a broadcast communication unit 231, an audio reproduction unit 232, and a moving image reproduction unit 233. [ Or the like. The user input unit 250 includes at least one of a button 251, a microphone 252, a keypad 263, and an input device 264. [ The sensing unit 270 includes at least one of a proximity sensor 271, an illuminance sensor 272, and an attitude sensor 273. The image input unit 280 includes at least one of a first camera 281 and a second camera 282.

In an embodiment, the storage unit 290 may be provided with a voice database 291 (hereinafter also referred to as voice DB). The portable device 200 according to an embodiment of the present invention may further include at least one of a mobile communication unit 225, a speaker 271, a vibration motor 272, a GPS unit 273, and a connector 275 have.

The control unit 210 stores at least one processor 211, a ROM 212 storing a control program for controlling the portable device 200 and signals or data input from the outside of the portable device 200, And a RAM (RAM) 213 used as a storage area for various operations performed in the portable device 200.

The control unit 210 controls the overall operation of the portable device 200 and the signal flow between the internal components 220 to 295 of the portable device 200 and performs a function of processing data. The control unit 210 controls power supply from the power supply unit 295 to the internal components 220 to 290. In addition, when the input or the set stored condition of the user is satisfied, the control unit 210 can execute an operating system (OS) and various applications stored in the storage unit 290. [

In one embodiment, the control unit 210 includes a direction analysis module for estimating a direction of a voice signal in accordance with various operations to be performed, a person recognition module for recognizing a human shape from image data, a voice for separating a voice signal from the input sound, A speaker separation module for separating signals corresponding to a predetermined speaker from the voice signal, a speaker recognition module for determining a speaker corresponding to the separated voice signal using the voice database 291, An eye tracking module for tracking the eye of the user, and a noise removal module for removing noise from the input sound. At least one of the above modules may be included in the controller 110 of the hearing aid 100.

The processor 211 may include at least one general purpose processor such as a CPU (Central Processing Unit), AP (Application Processor), Microcomputer (MICOM) It is possible to implement various operations of the portable device 200 by loading and executing a corresponding program in the RAM 213. [

The processor 211 may include single core, dual core, triple core, quad core, and cores thereof. A processor may include a plurality of processors, for example, a main processor and a sub processor. The sub processor is provided to operate in a standby mode (hereinafter also referred to as a sleep mode) in which only standby power is supplied and does not operate as the portable device 200. [

The processor 211, the ROM 212, and the RAM 213 may be interconnected via an internal bus.

In one embodiment, the processor 211 may include a graphics processing unit (GPU) (not shown) for graphics processing. For example, when the portable device 200 is implemented as a smart phone, a smart pad, or a smart TV, the processor 211 may be a SoC (System On) system, which is a combination of a core (not shown) and a GPU (not shown) Chip).

In another embodiment, the control unit 210 may further include a separate GPU for graphics processing.

Meanwhile, in another embodiment of the present invention, the control unit 210 may include a program for performing a function of detecting an error occurrence in a specific configuration including a main processor, for example, a specific function supported by the portable device 200 For example, an integrated circuit (IC) chip provided as a dedicated processor for executing the program.

The control unit 210 includes a communication unit 220, a mobile communication unit 225, an image processing unit 230, a display unit 240 including a touch screen 241, a touch screen controller 245, a user input unit 250, A speaker unit 271, a vibration motor 272, a GPS unit 273, an image input unit 280, a storage unit 290, and a power supply unit 295, as shown in FIG.

The communication unit 220 may include at least one of a wireless LAN unit 221 and a local communication unit 222. For example, it may include only the wireless LAN unit 221, only the short range communication unit 222, or both the wireless LAN unit 221 and the short range communication unit 232.

The wireless LAN unit 231 may be connected to an access point (AP) by using wireless in a place where an access point (AP) (not shown) is installed, under the control of the controller 210. [ The wireless LAN unit 231 supports the IEEE 802.11x standard of the Institute of Electrical and Electronics Engineers (IEEE). In addition, the short-range communication unit 232 can perform short-range communication with other devices including the hearing aid device 100 wirelessly without an access point (AP) under the control of the controller 210. [ Near-field communication can be achieved through Bluetooth, bluetooth low energy, IrDA, Wi-Fi, Wi-Fi Direct, Zigbee, UWB Ultra Wideband), Near Field Communication (NFC), and the like.

In an embodiment, the communication unit 220 may be configured to support human body communication using a magnetic field.

The mobile communication unit 225 can be connected to an external device, i.e., another device, by mobile communication using one or more antennas (not shown) under the control of the controller 210. [ The mobile communication unit 225 is connected to the portable device 200 through a voice communication, a video call, a text message (SMS), a multimedia message (MMS), or the like with a mobile phone, a smart phone, a tablet, And a wireless signal for data communication.

The portable device 200 may include at least one of a mobile communication unit 225, a wireless LAN unit 221, and a local communication unit 222 according to performance. For example, the portable device 200 may include a combination of two or more of the mobile communication unit 225, the wireless LAN unit 221, and the local communication unit 222 depending on the performance.

The communication unit 220 according to the embodiment of the present invention may be connected to another device (for example, the hearing aid 100) by the control of the control unit 210 to transmit / receive data. The transmission / reception of such data includes transmission and reception of audio signals and video data.

In this embodiment, the term communication unit basically indicates the communication unit 220, but it is defined as including both the communication unit 220 and the mobile communication unit 225 in some cases.

The image processing unit 230 may include a broadcast communication unit 231, an audio reproducing unit 232, or a moving image reproducing unit 233. [ The broadcast communication unit 231 may transmit a broadcast signal (for example, a TV broadcast signal, a radio broadcast signal, or a data broadcast signal) transmitted from an external broadcast station through a broadcast communication antenna (not shown) ) And the broadcasting unit may receive information (for example, Electric Program Guide (EPS) or Electric Service Guide (ESG)). In addition, the control unit 210 processes the received broadcast signal and the broadcast unit so that the information can be played back on the display unit 240 and / or the speaker 271 using a video codec unit and / or an audio codec unit .

The audio reproduction unit 232 is connected to an audio source (for example, mp3, wma, etc.) stored in the storage unit 290 of the portable device 200 or received from the outside, under the control of the control unit 210. [ ogg, or wav) can be reproduced on the speaker 271. [0100]

The audio reproduction unit 232 of the embodiment of the present invention controls the auditory feedback (for example, the storage unit 290) corresponding to the continuous movement of the touch or touch detected on the touch screen 241 under the control of the control unit 210, Output of an audio source stored in the audio codec unit) through the audio codec unit.

The moving picture reproducing unit 233 may be a digital moving picture source which is stored in the storage unit 290 of the portable device 200 or received from outside according to the control of the controller 210. For example, mp4, avi, mov, or mkv) can be played back using a video codec unit. Most of the applications that can be installed in the portable device 200 can reproduce an audio source and / or a moving picture file using an audio codec unit and / or a video codec unit.

The moving picture reproducing unit 233 may generate a visual feedback based on the continuous movement of the touch or touch detected by the touch screen 241 under the control of the controller 210 290) can be reproduced through the video codec unit.

In addition, it will be readily understood by a person skilled in the art that many kinds of video and audio codec units are produced and sold.

The image processing unit 230 may include an audio reproduction unit 232 and a moving picture reproduction unit 233 except for the broadcast communication unit 231 in correspondence with the performance or structure of the portable device 200. [ The audio reproduction unit 232 and / or the moving picture reproduction unit 233 of the image processing unit 230 may be included in the control unit 210. In an embodiment of the present invention, the term video codec unit may include one or more video codec units. Further, in the embodiment of the present invention, the term audio codec unit may include one or two or more audio codec units.

The display unit 240 displays an image based on the image signal processed by the image processing unit 230. The method of implementing the display unit 240 is not limited. For example, a liquid crystal, a plasma, a light-emitting diode, an organic light-emitting diode, electron conduction electron-emitter, carbon nano-tube, nano-crystal, and the like.

The display unit 240 may further include an additional configuration depending on the implementation method. For example, when the display unit 240 is a liquid crystal type, the display unit 240 includes a liquid crystal display panel (not shown), a backlight unit (not shown) for supplying light thereto, and a panel (Not shown).

In one embodiment, the display 240 may include a touch-screen 241 on which an input according to a user's touch is received. The touch screen 241 may be implemented by, for example, a resistive method, a capacitive method, an infrared method, or an acoustic wave method.

The touch screen 241 may display an object (e.g., a menu, text, image, video, graphic, icon, and shortcut icon) containing a menu item of the display device 100 as a user interface have. The user can touch the object displayed on the touch screen 241 with a separate pointing device 254 such as a user's body (e.g., a finger) or a star illustration to perform user input.

The touch screen 241 may provide a GUI (Graphical User Interface) corresponding to various services (for example, call, data transmission, broadcasting, photographing, moving picture, or application) to the user. The touch screen 241 transmits an analog signal corresponding to a single touch or multi-touch input through the GUI to the control unit 110. [

In one embodiment, the touch is not limited to the contact of the touch screen 241 with the user's body or a touchable pointing device, but may be a non-contact (e.g., touch screen 241 and pointing with the user's body or touch screen 121) Hovering with a detectable spacing of 30 mm or less with the device). It will be readily understood by those skilled in the art that the noncontact interval that can be detected by the touch screen 241 can be changed according to the performance or structure of the portable device 200. [

In this embodiment, the touch input includes a drag, a flick, a drag and drop, a tap, a long tap, and the like.

The user input unit 250 may include at least one of one or more buttons 251, a microphone 252, a keypad 253, and an input device 254.

The button 251 includes at least one of a menu button, a home button, and a back button provided at the bottom of the front side. The button 251 may include a side power / lock button and / or a volume button. Here, in the portable device 200, the button 251 may be implemented as a touch button provided on the touch screen 241 instead of a physical button. In the portable device 200, the button 251 may be displayed as text or an icon in the touch screen 241.

In one embodiment, the user input unit 250 of the portable device 200 may receive user input in response to an interaction with the content displayed on the touch screen 241, that is, a touch or a touch gesture.

A microphone 252 receives voice or sound from the outside under the control of the controller 210 and generates an electrical signal. The electrical signal generated by the microphone 252 may be converted in the audio codec unit and stored in the storage unit 290 or output through the speaker 271. [ The microphone 252 may be located on one or more of the front, side, and rear surfaces of the portable device 200.

The keypad 253 may receive a key input from a user for control of the portable device 200. [ The keypad 253 includes a physical keypad (not shown) formed on the front surface of the portable device 200, a virtual keypad (not shown) displayed in the touch screen 241, and an external keypad And a keyboard dock (not shown). When the keypad 253 is an external keypad, the key input signal of the user is received through the communication unit 120 or the connector 275 .

It will be easily understood by those skilled in the art that the physical keypad formed on the front surface of the portable device 200 can be excluded depending on the performance or structure of the portable device 200. [

The input device 254 may touch an object (e.g., a menu, a text, an image, a video, a graphic, an icon, and a shortcut icon) displayed on a screen displayed on the touch screen 241 of the portable device 200 You can choose. The input device 254 can touch or select content as an object displayed on the touch screen 241 of the portable device 200. [ In the present embodiment, the contents are stored in the portable device 200, such as an application installed in the portable device 200, video data photographed through the cameras 261 and 262, media data stored in the storage unit 190, text data such as documents and contacts, And various data that can be transmitted and received.

The input device 254 can input characters, for example, by touching a touch screen of a capacitive type, a resistance type, and an electromagnetic induction type, or by using a virtual keyboard. The input device 254 may include a built-in pen vibration element (e.g., a vibration motor or actuator) using the control information received from the communication unit 220 of the pointing device, the stylus, A haptic pen or the like which vibrates. Further, the vibration element may vibrate using the sensing information detected by a sensor (for example, an acceleration sensor, not shown) built in the input device 254 instead of the control information received from the portable device 200 . The input device 254 can be inserted into the insertion port of the portable device 200 and can be omitted depending on the performance or structure of the portable device 200. For example, .

The sensing unit 260 includes at least one sensor for detecting the state of the portable device 200. [ For example, the sensing unit 260 may include a proximity sensor 261 located on one side of the front of the user's portable device 200 and detecting whether the portable device 200 is approaching the portable device 200, An attitude sensor 262 for detecting the direction of gravity, a posture sensor 263 for detecting the posture and direction of the portable device 200, a gravity sensor for detecting the direction of action of gravity, And an altimeter for detecting the altitude. The attitude sensor 263 may include a gyro sensor using rotary inertia, an acceleration sensor for detecting inclination of three axes (for example, x axis, y axis, z axis) applied to the portable device 200 .

The sensing unit 260 can measure the acceleration in which the motion acceleration of the portable device is added to the gravitational acceleration, and can measure only the gravitational acceleration in the case where the portable device 260 does not move. For example, when the front face of the portable device 200 faces upward, the gravitational acceleration is in the positive direction, and when the back face of the portable device 200 faces upward, the gravitational acceleration may be in the negative direction .

At least one sensor included in the sensing unit 260 detects the state of the portable device 200, generates a signal corresponding to the sensing, and transmits the signal to the control unit 210.

It will be readily understood by those skilled in the art that the sensor of the sensing unit 260 can be added or deleted depending on the performance of the portable device 200. [

The speaker 271 receives various signals (for example, wireless signals) from the communication unit 220, the mobile communication unit 225, the image processing unit 230, or the image input unit 280 using the audio codec unit under the control of the control unit 210. [ A sound, a video signal, a picture signal, a broadcast signal, an audio source, a moving picture file, or a picture) to the outside of the portable device 200.

The speaker 271 can output a sound corresponding to the function performed by the portable device 200 (for example, a touch operation sound corresponding to a telephone number input, a photographing button operation sound, etc.). At least one speaker 271 may be positioned on the front, side, and back of the portable device 200.

In an embodiment of the present invention, the speaker 271 may output auditory feedback corresponding to the continuous movement of the touch or touch detected by the touch screen 241 under the control of the control unit 210. [

The vibration motor 272 can convert the electrical signal into mechanical vibration under the control of the control unit 210. [ For example, the vibration motor 272 may include a linear vibration motor, a bar type vibration motor, a coin type vibration motor, or a piezoelectric element vibration motor. When a voice call request is received from another portable device, the vibration motor 266 of the portable device 200 in the vibration mode operates under the control of the control unit 210. [ The vibration motor 266 may be located on the portable device 200 in one or more than two. In addition, the vibration motor 266 can cause the entire portable device 200 to vibrate or vibrate only a part of the portable device 200. [

The GPS unit 273 periodically acquires information (e.g., accurate position information of GPS satellites (not shown) that can be received by the portable device 200) from the plurality of GPS satellites (not shown) Information). The portable device 200 can know the position, speed, and / or time of the portable device 200 using information received from a plurality of GPS satellites.

The connector 275 can be used as an interface for connecting the portable device 200 to an external device (not shown) or a power source (not shown). The portable device 200 can transmit data stored in the storage unit 290 to the external device or receive data from the external device through the cable connected to the connector 275 under the control of the control unit 210. [ The portable device 200 can charge the battery from the power source via the wired cable connected to the connector 275 or from the battery provided as the power supply 295. [ In addition, the portable device 200 can be connected to an external accessory (e.g., a photo printer, not shown) through the connector 275. [

The image input unit 280 may include at least one of a first camera 281 on the front surface and a second camera 282 on the rear surface for capturing still images or moving images under the control of the controller 210. [ The image input unit 280 may include one or both of the first camera 281 and the second camera 282. The image input unit 280 may further include an auxiliary light source (for example, a flash 283) that provides an amount of light necessary for taking a picture of the first camera 281 or the second camera 282.

The storage unit 290 includes a communication unit 220, a mobile communication unit 225, an image processing unit 230, a display unit 240, a user input unit 250, a sensing unit 260, And may store signals or data input / output corresponding to the operation of the operation unit 271, the vibration motor 272, the GPS unit 273, the connector 275, and / or the image input unit 280. The storage unit 290 stores a control program for controlling the portable device 200, a graphical user interface (GUI) related to an application downloaded from an external source or provided by a manufacturer, images for providing a GUI, user information, Bases or related data.

In one embodiment, an application for interlocking with the hearing aid device 100 may be installed in the portable device 200, and the application and related data are stored in the storage unit 290.

In one embodiment, the storage unit 290 may store touch information corresponding to a continuous movement of a touch and / or a touch (e.g., X and Y coordinates of the detected touch position, touch detection time, etc.). The storage unit 290 may store the type of continuous movement of the touch (e.g., flick, drag, or drag and drop), and the control unit 210 may compare the inputted user touch with the information of the storage unit 290 So that the type of touch can be identified. The storage unit 290 stores visual feedback (for example, a video source or the like) recognizable by a user corresponding to the input touch or touch gesture and output to the display unit 290, (E. G., A haptic pattern, etc.) output from the auditory feedback (e. G., A sound source, etc.) and the vibration motor 272.

The storage unit 290 according to an embodiment of the present invention is provided with a voice database 291. In the voice database 291, information on the hearing loss frequency band of a wearer who is a user of the hearing aid device 100, that is, a hearing loss, can be stored.

The voice database 291 may store information corresponding to a plurality of speakers. A plurality of speakers in which information is stored in the voice database 291 include surrounding persons such as family members and friends who mainly talk with the wearer of the hearing apparatus 100. Information to be stored includes information of voice signals for each speaker (for example, Frequency band, tone, tempo, etc.) and user information (e.g., image, sex, age, name, etc.). Here, the user information may be input through the user input unit 250 or may be received from the outside through the communication unit 220. [ In addition, the image may be an image photographed through the image input unit 280.

The information stored in the voice database 291 is used to identify a speaker corresponding to the received voice signal and is updated with a time at which information of a new speaker is added. That is, if the speaker information corresponding to the received voice signal is not present in the voice database 291, the controller 210 may update the voice database 291 with the voice information.

The control unit 210 may request to input information about the speaker in the information storage process. That is, the control unit 210 may display on the display unit 240 a UI requesting the wearer of the portable device 200, that is, the wearer of the portable device 200, to input information about the speaker, (E.g., name, gender, relationship (family or friend)) by using the user input unit 250 in response to the request. The control unit 110 may cause the video database unit 291 to store the video shot through the video input unit 280 or the video stored in the storage unit 290 in correspondence with a specific speaker in response to a user's operation.

In one embodiment, the voice database 291 includes not only information on the frequency domain of the voice signal received from the voice input unit 160 but also information on the interaural time difference (ITD), interaural level difference (ILD) arrival) can be further stored.

In one embodiment of the present invention, the hearing aid device 100 and the portable device 200 have the same user account. That is, the wearer of the hearing aid apparatus 100 downloads and installs an application for controlling the hearing aid apparatus 100 in the portable device 200, and can log in to the application using his / her user account.

4 shows an example in which the voice database 291 is provided in the portable device 200. The present invention can be applied to the case where the voice database is provided in the hearing aid 100 or the hearing aid 100 ) And / or an external device (e.g., a cloud server) capable of communicating with the portable device 200. The cloud server may store information on a registered user corresponding to the logged-in user account, that is, information on the wearer of the hearing aid device 100, and information on a plurality of speakers that can be conversation partners.

In one embodiment of the present invention, the portable device 200 can be implemented as a wearable device that can be worn on the body, for example, a smart watch. 5 is a perspective view illustrating a portable device 200 implemented as a smart watch according to an embodiment of the present invention.

The control unit 210 of the portable device 200 according to an embodiment of the present invention receives the voice signal from the hearing device 100 by the communication unit 220 and outputs the direction Can be determined. In this process, location tracking algorithms such as time difference of arrival (TDOA) and delay-and-sum beam forming may be used.

6 and 7 are diagrams for explaining a method of estimating the position of a speaker in the portable device 200 according to the embodiment of the present invention.

6, a microphone array including a plurality of microphones 161 and 162 operated as directional microphones is provided as a voice input unit 160 in the hearing aid apparatus 100. As shown in FIG.

6 shows an example of using the TDOA algorithm using the time difference of the sound waves reaching the respective microphones 161 and 162. [

6, it is assumed that a sound source located at a specific point, that is, a sound generated from a speaker, is input to the two microphones 161 and 162 in a planar manner. Here, sound (sound waves) first arrive at the first microphone 161 closer to the sound source, that is, the speaker, and arrives at the second microphone 162 late by the arrival delay time.

The direction of the sound source can be obtained by calculating the angle between the two microphones 161 and 162 and the sound source. The difference? S between the sound wave traveling distance from the sound source to the first microphone 161 and the sound wave traveling distance from the sound source to the second microphone 162 can be expressed by Equation 1 below.

Here, τ represents the arrival delay time (time constant), v represents the speed of the sound wave, and d represents the separation distance between the first microphone 161 and the second microphone 162.

Accordingly, the following equation (2) is established.

Accordingly, it is possible to estimate the direction of the sound source, that is, the position of the speaker, using Equation (2).

6 can be applied to a three-dimensional space by increasing the number of microphones included in the microphone array by applying the basic principle described in FIG. Further, if a sufficient number of microphones is secured, it is possible to estimate not only the direction of the sound source in the three-dimensional space but also the position of the sound source (i.e., the distance to the sound source).

FIG. 7 shows an example using a delay-and-sum beam forming algorithm.

For example, as shown in FIG. 7, a microphone array including four microphones 161, 162, 163, and 164 in the hearing apparatus 100 is provided as a voice input unit 160, Assuming that a sound wave has occurred in a sound source corresponding to an arbitrary position, the output signal (output) thereof can be expressed by Equation (3).

Where l is the distance between the microphones (in meters), f is the frequency, θ is the angle with which the sound waves arrive, c is the speed of the sound wave (eg 343 m / s), N is the number of microphones Respectively.

Accordingly, as shown in FIG. 6, the speech signal received through each of the microphones 161, 162, 163, and 164 can be applied to Equation (3) to estimate the sound source, i.e., the direction (position) of the speaker.

The control unit 210 of the portable device 200 determines whether the other party of the conversation exists in the speech position (sound source position) of the speech signal estimated using the image data photographed by the image input unit 180 of the hearing aid apparatus 100 Can be determined. Here, the firing position is the position of the wearer's right, left, up, down, center, left up, right up, Down, Right Down, and the like. The division of the firing position is described by way of example. In some cases, it may be simplified to the right, left, upper, and down or may be further divided.

The ignition position information classified as described above may be stored in advance in the storage unit 190 as a kind of matching table. In one embodiment, the voice database 291 may include a matching table.

Here, the image data includes the wearer's front image data photographed by the front camera 181, and when the front image data includes the human shape at the estimated ignition position, the controller 210 determines that the speaker exists at the ignition position .

In one embodiment, when the image input unit 180 further includes a rear camera 182 that photographs a wearer's eye region, the controller 210 uses the eye information of the rear image data to determine whether the wearer looks at the corresponding speaker Can be determined.

8 is a diagram for explaining a method of using eye information in the portable device 200 according to an embodiment of the present invention.

In one embodiment, the ocular information can be determined using the up, down, left, and right lengths (L1 to L8) of the outside of the eyeball as ocular direction information as shown in Fig. For this purpose, the matching directions provided by L1 to L8 may be matched and stored in the matching table provided in the storage unit 290. For example, if L1 and L5 are longer than the reference, L2 and L6 are shorter than the reference, and L3, L4, L7, and L8 are within the reference range, the eye direction is matched to the right. In the same manner, if L1 and L5 are shorter than the reference, L2 and L6 are longer than the reference, L3 and L7 are longer than the reference, and L4 and L8 are shorter than the reference, the eye direction is matched to the lower left.

The control unit 210 obtains the values of L 1 to L 8 from the rear image data through the rear camera 182 of the hearing aid 100. The control unit 210 loads the eye direction matched to the obtained L1 to L8 from the matching table to determine the eye direction of the user (wearer of the hearing aid).

In another embodiment, tracking in the eye direction may be performed by the control unit 110 of the hearing aid 100. [

In the embodiment of the present invention as described above, the user's eyeball direction is determined using the lengths L1 to L8. However, the present invention is not limited to this, and various algorithms known to track eyeballs It will be readily understood by those skilled in the art.

In an embodiment of the present invention, the control unit 210 of the portable device 200 can remove noise from the received voice signal. Various noise cancellation algorithms can be used to remove noise, including a method for separating voice signals and ambient noise signals.

In another embodiment, noise removal may be performed by the control unit 110 of the hearing aid 100. [ In this case, a signal from which the noise signal is removed from the voice signal is transmitted to the portable device 200 through the communication unit 120.

The portable device 200 according to the embodiment of the present invention uses the voice signal and the video data received through the hearing aid device 100 to determine whether or not the wearer (listener) of the hearing aid device 100 is speaking So that the listener can concentrate on the other party of the conversation without confusion.

In an exemplary embodiment of the present invention, an application provided for the hearing impaired user, which is a software configuration driven by at least one processor 110, 210, which is a hardware configuration, may be installed in the hearing aid 100 and / or the portable device 200 have. That is, the operations described below can also be performed according to the execution of the software driven by the processors 110 and 210 provided in the controllers 100 and 200.

Accordingly, the control units 100 and 200 may include a plurality of hardware or software modules, and each of the modules may include a voice direction analysis module, a human recognition module, a voice signal determination module, a speaker separation module, A speaker recognition module, an eye tracking module, a noise removal module, and the like. In this embodiment, the operations performed by the respective modules are defined as being performed by the control units 100 and 200 as well.

Hereinafter, a related embodiment will be described in detail with reference to Figs. 9 to 12. Fig.

Hereinafter, a control method in the hearing aid system 1 including the hearing aid device 100 and the portable device 100 according to the embodiments of the present invention will be described with reference to the drawings.

FIG. 9 is a flowchart illustrating a control method of the hearing aid system 1 according to an embodiment of the present invention. FIGS. 10 to 12 are flowcharts illustrating a method of controlling the hearing aid apparatus 100 and the portable device 200 according to the control method shown in FIG. FIG. 2 is a diagram illustrating a process of conversation by a person with a hearing impairment using the other party.

9 to 12, a user 301 who is a hearing impaired person can wear a hearing aid apparatus 100 and talk with a large number of people including the other party 302. [ The present embodiment determines whether the user 301 is in a conversation in this process, and determines whether or not a speaker is present at the utterance position of the voice signal and / (302). By performing corresponding actions, the user 301 who is a hearing impaired person concentrates on the voice of the other party 302, which is the party of the current conversation, among the plurality of people, and amplifies the voice of the persons other than the other party So that it can be excluded from the object.

9 and 10, when the user wears the hearing aid device 100 and the power of the hearing aid device 100 and the portable device 200 is turned on, A signal and image data may be received (S902).

Here, the audio signal is received through a plurality of microphones 161, 162, and 163 provided as the audio input unit 160 of the hearing aid apparatus 100, and the video data is provided as the video input unit 180 of the hearing aid apparatus 100 The front camera 181 can be used as a front camera. Here, when the hearing aid 100 further includes the rear camera 182, not only the front image data photographed by the front camera 181, but also the rear image data are further received by the rear camera 182, The image data may include eyeball information of a user, that is, eyeball direction information.

In one embodiment, the voice signal and the image data received in step S902 are transmitted to the portable device 200 through the communication unit 120 of the hearing aid. In step S902, various sounds including a voice signal may be received, and the control unit 210 may selectively extract a voice signal from the received voice using the voice database 291. [

The portable device 200 receives the voice signal and the image data transmitted in step S902 through the communication unit 220 and the control unit 210 transmits the voice signal and the image data to the user 100 (Step S904). For example, the control unit 210 may recognize the human shape from the front image data, and determine that the image data is in conversation when a person is included in the image data.

If it is determined in step S904 that the user is not in conversation, the process proceeds to step S922, which will be described later, so that the level of the hearing loss frequency range of the speech signal can be amplified and output through the sound output unit 170, Noise can be removed from the signal.

The control unit 210 may determine the speech position of the speech signal received in step S902, for example, the direction of the sound wave (S906). In this process, the position estimation algorithm described in Fig. 6 or 7 can be used. The determined firing position in step S906 is the user's right, left, top, down, center, left up, right up, And can correspond to any one of a left down (Left Down) and a right down (Right Down).

The control unit 210 may determine whether there is a speaker at the ignition position determined in step S906 using the front image data (S908). Specifically, when the ignition position is determined in step S906, it is determined that the speaker exists at the ignition position when the human shape is photographed at the corresponding position of the front image data. Here, the controller 210 may determine that the speaker exists at the utterance position when the speaker position estimated using the utterance position and the image data coincide or the error exists within the predetermined reference range.

In one embodiment, the sensing value of the direction sensor 165 may further be used in the determination process of step S908. For example, the control unit 210 may utilize the sensed value of the direction sensor 165 to calculate a direction corresponding to a human shape estimated as a conversation partner and use the calculation result. The sensing value of the direction sensor 165 is transmitted to the portable device 200 through the communication unit 120.

The controller 210 determines the eye direction of a user wearing the hearing aid apparatus 100 using the rear image data, and further determines whether the user answers the speaker based on the determined eye direction (S910). In this process, the eye tracking algorithm described in Fig. 8 can be used. That is, when the speaking position determined in step S906 matches the eye direction determined in step S910, the control unit 210 can determine that the user gazes at the speaker.

If it is determined in step S908 and step S910 that a speaker is present at the utterance position of the voice signal and the user looks at the speaker, the controller 210 determines the speaker as the conversation partner (S910).

The control unit 210 can determine whether or not the information of the speaker determined as the conversation partner exists in the speech database 291 (S914). In this process, the control unit 210 separates the voice signal corresponding to the specific speaker by using the information (e.g., frequency band, tone, tempo, etc.) of the speaker stored in the voice database 291 And a series of voice signal processing processes such as removing noise from the separated voice signal can be performed. The speech signal processing process may further include separating the speech signal from the input sound.

If it is determined in step S914 that the speaker information is present in the voice database 291, the control unit 210 determines a speaker corresponding to the received voice signal using the voice database 291 (S916). Here, the control unit 210 can determine the name of the speaker, the sex, and the relationship with the user.

If it is determined in step S914 that the speaker information does not exist in the voice database 291, the speaker database 291 may be stored with the speaker information corresponding to the received voice signal (S918). Here, image data photographed through the front camera 181, information on a speaker (e.g., name, sex, relationship with a user, etc.) that can be input by the user input unit 250 is matched with a voice signal Lt; / RTI >

In one embodiment, the process of steps S914 to S918 may be performed even if the speaker is not present at the firing position at step S908, or if the wearer is determined not to stare at the speaker at step S910.

The results of steps S904 to S916 as described above may be displayed by the display unit 140 of the hearing aid apparatus 100 (S920), as shown in FIG. 11 or FIG. To this end, the control unit 210 transmits the determination result in steps S904 to S916 to the hearing aid 100 through the communication unit 220. [ The control unit 110 of the hearing aid apparatus 100 can receive the result through the communication unit 120 and control the image processing unit 130 to display the corresponding image.

For example, if it is determined in step S904 that the wearer of the hearing aid apparatus 100, that is, the user is in conversation, the text 401 indicating the conversation mode may be displayed on the display unit 140 as shown in Fig. In addition, a predetermined icon 402 may be displayed on the display unit 140 at the position corresponding to the ignition position determined in step S906. FIG. 11 shows an example where the ignition position is centered on the center.

As another example, the information of the speaker determined in step S916 may be displayed by the display unit 140 as shown in FIG. (For example, name, gender, and the like) existing in the voice database 291 as a speaker corresponding to the received voice signal and displays a text 403 indicating the user on the display unit 140 do.

In addition, when the gender is determined to be female as shown in FIG. 12, the location of the utterance can be displayed by the icon 404 that can identify the sex. Here, when the ignition position is the front left, the position of the icon 405 is moved correspondingly.

11 and 12 may be stored in advance in the storage unit 190 of the hearing aid apparatus 100 or the storage unit 290 of the portable device 200 and may be stored in the external Lt; / RTI > server (e. G., A cloud server). In addition, the type of the text or the icon can be added by a user input or an administrator.

Then, in step S922, the speech output unit 170 of the hearing aid 100 outputs the speech signal whose level of the hearing loss frequency region is increased with respect to the speech signal received in step S902. Here, the output speech signal may be a speech signal separated in accordance with a specific speaker and / or a speech signal in which noise is removed in step S914. For example, based on the information stored in the voice database 291 for the speaker determined as the other party in step S916, if the corresponding voice signal is present in the frequency band of the user's hearing loss area, the level of the voice signal is increased You can do it.

Accordingly, by separating the voice signal estimated to be the other party of the conversation and outputting the increased level of the hearing loss frequency region with respect to the separated voice signal, the wearer of the hearing aid 100 can concentrate on the conversation with the other party even in a noisy environment do.

It is to be understood that each of the features of the various embodiments of the present invention may be combined or combined with each other partially or entirely and technically various interlocking and driving is possible as will be appreciated by those skilled in the art, It may be possible to cooperate with each other in association.

Thus, according to the embodiment of the present invention, the hearing aid 100 uses the audio signal received through the plurality of microphones 161, 162, and 163 and the video data received through the front camera 181, And the speaker present at the uttered position is specified as the other party of the conversation, and the voice signal of the specified counterpart is selectively amplified and output, so that the user who is the hearing impaired speaker discomforts the voice of the conversation partner in any environment It will be possible to listen without it.

Also, by allowing the user to determine whether or not to look at the speaker at the utterance position via the rear camera 182, it is possible to accurately specify the other party of the conversation even in a noisy environment in which there are many speakers, Can be improved.

Accordingly, it is possible to provide an improved hearing aid system which can help a user wearing the hearing aid apparatus 100 to listen to and understand a necessary voice by just looking at the other party.

In addition, since the information of the voice database 291 stored in correspondence with the frequent conversation partners is used, accuracy of the voice database 291 is improved, and the voice database unique to the user can be configured through the process of continuously updating the information.

Meanwhile, various embodiments of the present invention as described above can be embodied as a computer-readable recording medium. A computer-readable recording medium includes a transmission medium and a storage medium for storing data readable by the computer system. The transmission medium may be implemented over a wired or wireless network in which computer systems are coupled together.

Various embodiments of the present invention may be implemented by a combination of hardware, hardware, and software. As hardware, the processor 200 may include a non-volatile memory in which a computer program, which is software, is stored, a RAM into which a computer program stored in a non-volatile memory is loaded, and a CPU that executes a computer program loaded in the RAM. Non-volatile memory includes, but is not limited to, a hard disk drive, flash memory, ROM, CD-ROMs, magnetic tapes, floppy disks, optical storage, . The non-volatile memory is an example of a computer-readable recording medium on which a computer-readable program of the present invention is recorded.

The computer program is a code that can be read and executed by the CPU, and includes a code for causing the control unit 110 and 210 to perform operations such as steps S902 to S922 shown in Fig.

The computer program may be embodied in software including an operating system or an application provided in the hearing aid 100 or the portable device 200 and / or software that interfaces with an external device.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.

1: hearing aid system 100: hearing aid device
110: control unit 120: communication unit
130: image processor 140:
150: user input unit 160: voice input unit
170: audio output unit 180: video input unit
190: storage unit 195: power supply unit
200: portable device 210: control unit
220: communication unit 230: image processing unit
240: display unit 250: user input unit
260: sensing unit 271: speaker
280: image input unit 290:
295: Power supply

Claims (20)

In the portable device,
A communication unit for receiving the audio signal and the video data from a hearing aid provided with a voice input unit including a plurality of microphones for receiving a voice signal and a video input unit for receiving video data;
A storage unit for storing a voice database;
Determining whether or not a speaker is present at the determined firing position using the received image data, and if it is determined that the speaker exists at the firing position, Adjusts the identified voice signal so that the level of the loudspeaker frequency region is increased with respect to the voice signal corresponding to the speaker identified using the voice database from the signal and transmits the adjusted voice signal to the hearing aid And at least one processor for controlling transmission to be transmitted,
And the voice signal adjusted to increase the level of the hearing loss frequency region is output by the hearing aid. The method according to claim 1,
Wherein the hearing aid is a body wearing device, and the image input unit includes a front camera installed on a front surface of the hearing aid,
Wherein the processor determines whether or not a speaker is present at the determined firing position using the image data photographed by the front face camera on the front face of the wearer's wearer. 3. The method of claim 2,
Wherein the image input unit further comprises a rear camera provided at a position through which at least one line of sight of the wearer's wearer passes and acquiring eye information of the wearer,
Wherein the processor further determines whether the wearer gazes at the speaker using the acquired eye information. 3. The method of claim 2,
The hearing aid is further provided with a direction sensor,
Wherein the processor controls the communication unit to receive the sensing value of the direction sensor from the hearing aid and further uses the received sensing value to determine whether a speaker exists in the determined speaking position . The method according to claim 1,
Wherein the processor estimates the speech position of the speech signal using at least one of a time difference of the speech signal reaching the plurality of microphones, a distance between the plurality of microphones, and a number of microphones. . 6. The method according to any one of claims 1 to 5,
Wherein the voice database stores information on a plurality of speakers and voice signals corresponding to the plurality of speakers,
Wherein the processor identifies and separates the speech signal of the speaker from the received speech signal based on the speech database and adjusts the level of the hearing loss frequency region to increase with respect to the speech signal of the separated speaker Lt; / RTI > The method according to claim 6,
Wherein the processor is configured to determine whether information corresponding to the received voice signal is present in the voice database and to store information about the voice signal in the database if the information is determined not to be present. Device. 8. The method of claim 7,
Further comprising a user input for receiving user input,
Wherein the processor stores information received through the user input unit in the voice database in association with the voice signal. In a hearing aid,
A voice input unit including a plurality of microphones for receiving voice signals;
A video input unit for receiving video data;
A communication unit for transmitting the received voice signal and image data to a portable device;
An audio output unit for outputting a voice signal;
The control unit controls the communication unit to receive the voice signal adjusted so that the level of the hearing loss frequency range is increased with respect to the voice signal corresponding to the predetermined speaker from the portable device and controls the voice output unit to output the adjusted voice signal Lt; / RTI > processor,
Wherein the image input unit includes a front camera installed on a front surface of the hearing aid, and it is determined whether or not a speaker is present at a speaking position of the voice signal using the image data photographed by the front face camera, The voice signal corresponding to the speaker identified using the voice database is adjusted and transmitted from the portable device to the hearing aid. 10. The method of claim 9,
Wherein the image input unit further comprises a rear camera provided at a position through which at least one line of sight of the wearer's wearer passes and acquiring eye information of the wearer,
Wherein whether or not the wearer gazes the speaker using the acquired eye information is further determined. 11. The method of claim 10,
And a display unit provided at a position through which at least one line of sight of the wearer passes,
Wherein the processor controls the display unit to display information on the speech position and information on a speaker identified using the speech database. A control method for a portable apparatus,
The method comprising: receiving the speech signal and the image data from a hearing aid provided with a speech input unit including a plurality of microphones for receiving speech signals and an image input unit for receiving image data;
Determining a speech position of the received speech signal;
Determining whether a speaker exists at the determined firing position using the received image data;
And adjusting the identified voice signal so that the level of the hearing loss frequency region is increased with respect to the voice signal corresponding to the speaker identified using the voice database from the received voice signal when it is determined that the speaker exists in the speaking position ;
And transmitting the adjusted voice signal from the portable device to the hearing aid so as to be outputted by the hearing aid. 13. The method of claim 12,
Wherein the hearing aid is a body wearing device, and the image input unit includes a front camera installed on a front surface of the hearing aid,
Wherein the step of determining whether or not the speaker exists determines whether or not a speaker is present at the determined ignition position using the image data photographed by the front camera on the front face of the wearer of the hearing aid Way. 14. The method of claim 13,
Wherein the image input unit further comprises a rear camera provided at a position through which at least one line of sight of the wearer's wearer passes and acquiring eye information of the wearer,
Further comprising the step of determining whether the wearer gazes at the speaker using the acquired ocular information. 13. The method of claim 12,
The step of determining the firing position may include estimating a firing position of the speech signal using at least one of a time difference of the speech signal reaching the plurality of microphones, a distance between the plurality of microphones, and a number of microphones Wherein the control method comprises: 16. The method according to any one of claims 12 to 15,
Wherein the voice database stores information on a plurality of speakers and voice signals corresponding to the plurality of speakers,
Further comprising identifying and separating the speech signal of the speaker from the received speech signal based on the speech database,
Wherein the step of adjusting the speech signal adjusts the level of the hearing loss frequency region to be increased with respect to the speech signal of the separated speaker. 17. The method of claim 16,
Determining whether information corresponding to the received voice signal is present in the voice database;
Further comprising the step of storing information on the voice signal in the database if it is determined that the information does not exist. A control method for a hearing aid system including a hearing aid and a portable device,
Receiving a voice signal from a plurality of microphones;
Receiving image data from a front face camera installed on a front face of the hearing aid;
Determining a speech position of the received speech signal;
Determining whether a speaker exists at the determined firing position using the received image data;
And adjusting the identified voice signal so that the level of the hearing loss frequency region is increased with respect to the voice signal corresponding to the speaker identified using the voice database from the received voice signal when it is determined that the speaker exists in the speaking position ;
And outputting the adjusted voice signal by the hearing aid. 19. The method of claim 18,
Further comprising the step of acquiring eye information of the wearer from a rear camera which is provided at a position through which at least one line of sight of the wearer's wearer passes and acquires eye information of the wearer,
Further comprising the step of determining whether the wearer gazes at the speaker using the acquired ocular information. 19. The method of claim 18,
Wherein the hearing aid further comprises a display unit at a position through which at least one line of sight of the wearer passes,
Further comprising the step of displaying, by the display unit, information on the firing position and information on a speaker identified using the speech database.

Images