JP6296646B2 - Hearing complement system, hearing complement device, and hearing complement method - Google Patents

Description

  The present invention relates to a hearing complement system, a hearing complement apparatus, and a hearing complement method that complement a user's hearing.

  2. Description of the Related Art Conventionally, a technique for solving the problem of hearing of a hearing impaired person using a hearing aid that amplifies sound is known. Recently, digital hearing aids that process digital signals of sounds and provide clear sounds with reduced noise to hearing-impaired people are becoming mainstream.

  For example, in Patent Document 1, an analog input sound signal is converted into a digital input signal, a program and setting values are read from an external memory, signal processing is performed on the digital input signal, and the signal processed signal is converted into an analog output signal. A technique for converting and switching an analog output signal or an analog input sound signal to output as an output sound signal is disclosed. There is also known a technique for reducing noise by dividing a low sound region and a high sound region and reducing a high sound region as necessary (see, for example, Patent Document 2).

  As described above, in the conventional hearing aid represented by Patent Documents 1 and 2 and the like, various ideas are made from the viewpoint of providing clearer sound to the hearing impaired person.

JP 2011-035684 A JP 2012-109933 A

  However, hearing-impaired people want to spend their daily lives in the same lifestyle as those with normal hearing who do not disturb their hearing as much as possible, and simply listen to the speaker's utterances with clear noise with reduced noise. I don't want it.

  For example, a normal hearing person often feels a sense of comfort and fulfillment due to life sounds such as a sound that can be heard from the kitchen. I cannot get a sense of fulfillment. Note that if the noise removal level is lowered so that the hearing impaired person can hear the life sound, the noise increases when the hearing impaired person is located in a public place, etc., and the hearing impaired person listens to clear sound. The result cannot be taken.

  In many cases, a normal hearing person inspects the speaker's heart from the tone indicating the high and low patterns of the speaker's voice and the voice color indicating the sound and tone of the voice, but the quantization error when quantizing digitally etc. If the information on tone and voice color is lost, the result is that a hearing-impaired person cannot perceive the speaker's heart like a normal hearing person.

  Furthermore, when a normal hearing person converses with multiple speakers, he or she unconsciously recognizes who is speaking in consideration of differences in the frequency of the voices of individual speakers and the direction of voice arrival. If a slight difference in direction information or frequency is lost due to the quantization error or the like, it is impossible to recognize who is speaking.

  From these facts, it is an important issue to complement the hearing gap between hearing-impaired and hearing-impaired and to provide a hearing-impaired person with a lifestyle equivalent to that of the hearing-impaired. Yes. Such a problem is not only a problem that occurs only for a hearing impaired person, but also a problem that occurs similarly for a user who is relatively deaf in hearing.

  The present invention has been made in order to solve the above-described conventional problems, and can efficiently complement the user's hearing and thereby provide a comfortable living environment for the user. An object of the present invention is to provide a hearing complement device and a hearing complement method.

In order to solve the above-described problems and achieve the object, the present invention is an auditory complement device that is worn on a user's ear and collects sound data that arrives at the user. means, a voice extracting means for extracting audio from collected sounds data by said sound collecting means, from the sound data collected by the said sound collecting means is used to complement the hearing of the user, Auditory supplement information extracting means for extracting auditory supplement information including at least life sound data , display control means for controlling display of the auditory supplement information extracted by the auditory supplement information extracting means on a predetermined display means, and at least the voice extraction Sound output means for outputting to the user output sound data obtained by superimposing the life sound data on the voice extracted by the means.

Further, in the above invention, the aural complementary information extraction means, a feature and Turkey to extract sound collected by a predetermined frequency band from the sound data as the auditory complementary information by the sound collecting means To do.

Furthermore, the present invention, in the above invention, further comprises a situation determination means for judging the surrounding situation of the user, wherein the auditory supplement information extracting means converts the sound data collected by the sound collecting means into the surrounding situation. the sound of the corresponding frequency band, wherein the benzalkonium be extracted as the aural complementary information.

Further, in the present invention according to the present invention, the auditory supplement information extracting unit extracts the arrival direction of the sound included in the sound data collected by the sound collector as the auditory supplement information, and the display control unit is The direction of arrival of the sound included in the sound data is controlled to be displayed on the display means .

  Further, the present invention is the above invention, further comprising a plurality of the sound collecting means, wherein the auditory complementary information extracting means is responsive to a difference in sound arrival times in the plurality of sound data collected by the plurality of sound collecting means. And determining the direction of arrival.

Also, in the present invention according to the above invention, the auditory supplement information extracting unit extracts a speaker's emotion as the auditory complement information from the voice included in the sound data collected by the sound collector, and the display control The means controls the display of the speaker's emotion on the display means .

  Also, in the present invention according to the above invention, the auditory supplement information extracting unit detects a change in the volume, height and / or pitch of the sound included in the sound data collected by the sound collecting unit. The emotion of the speaker estimated by the change is extracted as the auditory complement information.

In addition, the present invention is an auditory complement device that is worn on a user's ear, and is collected by one or more sound collecting means for collecting sound data arriving at the user, and the sound collecting means. extraction and voice extracting means for extracting a sound from the sound data from the sound data collected by the said sound collecting means is used to complement the hearing of the user, the auditory complementary information including at least living sound data Auditory supplement information extracting means , display control means for controlling the auditory supplement information extracted by the auditory supplement information extracting means on a predetermined display means, and at least the voice extracted by the voice extracting means in the living sound data And sound output means for outputting the output sound data superimposed with the user to the user.

The present invention also provides a sound collection step for collecting sound data arriving at a user, a sound extraction step for extracting sound from the sound data collected by the sound collection step, and a sound collection step by the sound collection step. from sounds data, the used to supplement the user's hearing, the hearing complementary information extracting step of extracting the aural complementary information including at least living sound data, hearing supplement extracted by the auditory complementary information extracting step A display control step for controlling display of information on a predetermined display unit, and a sound output step for outputting to the user output sound data in which the life sound data is superimposed on the sound extracted by at least the sound extraction step. It is characterized by that.

According to the present invention, the hearing a Subscriber efficiently complement, it is possible to provide a comfortable living environment for the user with.

FIG. 1 is an explanatory diagram for explaining a hearing complementation system according to the first embodiment. FIG. 2 is a block diagram showing an internal configuration of the earpiece shown in FIG. FIG. 3 is a block diagram showing an internal configuration of the core unit shown in FIG. FIG. 4 is an explanatory diagram for explaining processing of sound data by the core unit. FIG. 5 is an explanatory diagram for explaining selection of a filter. FIG. 6 is an explanatory diagram for explaining identification of a sound arrival direction. FIG. 7 is an explanatory diagram for explaining identification of a speaker's emotion. FIG. 8 is an explanatory diagram for explaining filtering on sound data. FIG. 9 is a flowchart showing a sound data processing procedure by the core unit. FIG. 10 is a block diagram illustrating a configuration of an earpiece that is a hearing complementation apparatus according to the second embodiment. FIG. 11 is a flowchart showing a processing procedure of sound data by the earpiece shown in FIG. FIG. 12 is an explanatory diagram for explaining the hearing complementation system according to the third embodiment. FIG. 13 is a block diagram showing the internal configuration of the earpiece shown in FIG. FIG. 14 is a block diagram showing an internal configuration of the core unit shown in FIG.

  Exemplary embodiments of a hearing complementation system, a hearing complementation apparatus, and a hearing complementation method according to the present invention will be described below in detail with reference to the accompanying drawings.

  First, the hearing complementation system according to the first embodiment will be described. FIG. 1 is a diagram illustrating a system configuration of an auditory complementation system. As shown in FIG. 1, the hearing complementation system includes an earpiece 10 and a core unit 20.

  The earpiece 10 is an ear hole type device that is worn by being inserted into the user's ear hole. The earpiece 10 is not limited to the ear hole type employed in the present embodiment, but may be an ear fold type to be put in an ear fold or an ear hook type to be hung on the ear.

  The core unit 20 is a main body device that performs processing related to hearing complementation according to the present invention, and is connected to the earpiece 10 so as to be communicable by short-range communication such as Bluetooth (registered trademark) or infrared rays. The core unit 20 can be stored in a user's pocket or mounted using a strap or the like.

  When the user does not use the earpiece 10 such as when the user sleeps, the earpiece 10 can be placed on a pod, a cradle, or the like and wirelessly charged without contact. Similarly, when the user does not use the core unit 20 such as when the user is sleeping, the core unit 20 can be placed on a cradle or the like and wirelessly charged without contact. The earpiece 10 can be wirelessly charged by the core unit 20 in a non-contact manner.

  The earpiece 10 and the core unit 20 each collect sound coming from the outside and generate sound data.

  The core unit 20 includes an auditory complement processing unit 23a that generates auditory supplement information from sound data collected by the own device. Specifically, the auditory complement processing unit 23a generates auditory supplement information including life sound data, sound direction data, and speaker emotion data from sound data collected by the own device.

  This living sound data is data obtained by extracting living sounds such as squeak sounds heard from the kitchen from sound data collected by the own device. The living sound data is generated by extracting the frequency and pattern of the living sound that is registered in advance from the sound data collected by the own device.

  The sound direction data is data indicating the arrival direction of the sound data collected by the own device. The speaker emotion data is data related to the emotion of the speaker extracted from the voice included in the sound data collected by the own device. When the tone or voice color of the speaker's voice changes over time, it is determined that the speaker's emotion has undulated, and this is extracted as speaker emotion data.

  The auditory complement processing unit 23a transmits life sound data to the earpiece 10 and controls display of sound direction data and speaker emotion data on the display unit 27a of the own device. FIG. 1 illustrates an example in which a speaker icon F1 indicating a speaker's emotion and a voice icon F2 indicating a voice arrival direction are displayed on the display unit 27a. For example, with the user's own position as the center of the display unit 27a, the arrival direction of the speaker's voice can be recognized from the relative position of the speaker icon F1 and the display angle of the voice icon F2, and the speaker icon F1. This makes it possible to guess the speaker's emotions.

  The earpiece 10 has a sound extraction unit 13a that extracts sound from a sound band included in sound data while removing noise from sound data collected by the own device. The sound extraction unit 13a is adjusted so that the sound can be effectively heard from the sound data collected by the own device.

  The earpiece 10 synthesizes and outputs the output sound data output from the sound extraction unit 13 a and the life sound data received from the core unit 20. For this reason, the user can hear not only the voice but also the living sound well, thereby recognizing the behavior of the family who performs the kitchen work, and can feel a sense of security and comfort.

  In addition, when the living sound data is not received from the core unit 20, the output sound data output from the sound extraction unit 13a is output as it is, so that the earpiece 10 operates as a single hearing aid. Note that it is also possible to output the speaker's emotion as a sound through the earpiece 10 using a tone signal or the like.

  Next, the internal configuration of the earpiece 10 shown in FIG. 1 will be described. FIG. 2 is a block diagram showing an internal configuration of the earpiece 10 shown in FIG. As illustrated in FIG. 2, the earpiece 10 includes a microphone 11, an AD conversion unit 12, a control unit 13, a wireless communication unit 14, a power supply unit 15, a DA conversion unit 16, and an earphone 17.

  The microphone 11 is a sound collecting unit that collects sound coming to the user and outputs it as an analog signal. The AD conversion unit 12 converts the analog signal output from the microphone 11 into a digital signal and outputs the digital signal to the control unit 13.

  The wireless communication unit 14 is a communication unit that performs wireless communication with the core unit 20. The wireless communication unit 14 receives sound data such as life sound data from the core unit 20 and outputs the sound data to the control unit 13.

  The control unit 13 is a DSP (Digital Signal Processor) that performs digital signal processing. The control unit 13 includes a voice extraction unit 13a and an output synthesis unit 13b therein. The voice extraction unit 13a is a voice extraction unit that filters noise components from the digital signal output from the AD conversion unit 12 and amplifies the frequency component corresponding to the voice to generate output sound data.

  The output synthesis unit 13b synthesizes the output sound data generated by the voice extraction unit 13a and the sound data received by the wireless communication unit 14 to generate synthesized sound data. When sound data is not received from the core unit 20, the output sound data generated by the sound extraction unit 13a and the synthesized sound data are the same.

  The DA converter 16 converts the synthesized sound data, which is a digital signal output from the controller 13, into an analog signal. The earphone 17 functions as a sound output unit that outputs a sound to the user's ear based on the analog signal output from the DA converter 16.

  The power supply unit 15 is a power supply that supplies power to the control unit 13 and the wireless communication unit 14, and the power supply unit 15 can be wirelessly charged by the pod, cradle, or core unit 20 as described above.

  When the earpiece 10 shown in FIG. 1 is used alone, it is attached to either the left or right ear, but the two earpieces 10 may be attached to the left and right ears, respectively.

  Next, the internal configuration of the core unit 20 shown in FIG. 1 will be described. FIG. 3 is a block diagram showing an internal configuration of the core unit 20 shown in FIG. As shown in FIG. 3, the core unit 20 includes a microphone 21a, a microphone 21b, an AD conversion unit 22a, an AD conversion unit 22b, a control unit 23, a wireless communication unit 24, a power supply unit 25, a storage unit 26, and a touch panel display 27. .

  The microphone 21a and the microphone 21b are sound collecting means for collecting sounds coming to the user and outputting them as analog signals. The microphone 21 a and the microphone 21 b are provided at spaced positions on the housing of the core unit 20. The AD conversion unit 22 a converts the analog signal output from the microphone 21 a into a digital signal and outputs the digital signal to the control unit 23. The AD conversion unit 22 b converts the analog signal output from the microphone 21 b into a digital signal and outputs the digital signal to the control unit 23.

  The wireless communication unit 24 is a communication unit that performs wireless communication with the earpiece 10 and an external device. The wireless communication unit 24 transmits the sound data output from the control unit 23 to the earpiece 10.

  The storage unit 26 is a storage device composed of a flash memory or the like. The storage unit 26 is used for storing control data related to the control of the core unit 20, recording data, and the like.

  The touch panel display 27 is a display operation unit that combines a display unit 27a including a liquid crystal display and an operation unit 27b including a touch panel. A physical button or the like may be provided as the operation unit 27b.

  The control unit 23 is a DSP (Digital Signal Processor) that performs digital signal processing. The control unit 23 includes an auditory complement processing unit 23a, an earpiece output processing unit 23c, a recording management unit 23d, a charge management unit 23e, and a communication management unit 23f.

  The auditory complement processing unit 23a is an auditory complement information extracting unit that generates auditory complement information that complements the user's hearing from sound data collected by the microphone 21a and the microphone 21b. Although the details will be described later, the auditory complement processing unit 23a outputs sound data such as life sound data among the generated auditory complement information to the earpiece output processing unit 23c, sound direction data, speaker emotion data, and the like. Is used for display control on the display unit 27a. That is, for the sound direction data and speaker emotion data, the display unit 27a is used as a notification unit. Further, since sound data such as life sound data is output to the earpiece 10 by the earpiece output processing unit 23c and finally output from the earphone 17, the earphone 17 functions as a notification unit.

  The earpiece output processing unit 23 c is a processing unit that outputs the sound data output by the auditory complement processing unit 23 a to the earpiece 10. Specifically, the earpiece output processing unit 23 c synthesizes sound data to be output to the earpiece 10, and transmits the synthesized sound data to the earpiece 10 by the wireless communication unit 24.

  The recording management unit 23d is a processing unit that stores sound data collected by the microphone 21a and the microphone 21b in the storage unit 26 as recording data, and reproduces or deletes the data in accordance with a user operation. It is preferable to always store the recorded data in the storage unit 26.

  The charge management unit 23 e is a processing unit that manages charging of the core unit 20 and the earpiece 10. The charge management unit 23e monitors the charge amounts of the core unit 20 and the earpiece 10, and notifies the user as necessary. In addition, the charge management unit 23 e identifies the cradle that has charged the core unit 20, and stores the cradle ID that identifies the cradle in the storage unit 26. The charge management unit 23e charges the earpiece 10 when the earpiece 10 is located within a predetermined distance.

  The communication management unit 23 f is a processing unit that manages the communication status of the wireless communication unit 24. The communication management unit 23f notifies the user when communication with the earpiece 10 is interrupted. In addition, when communication with the earpiece 10 is interrupted, the communication management unit 23 f stores the time information in the storage unit 26. If the position information can be acquired by GPS (Global Positioning System) or the like, a location where communication with the earpiece 10 is interrupted may be stored in the storage unit 26. By storing information at the time of the communication interruption with the earpiece 10 in this way, a clue for finding the earpiece 10 when the user loses the earpiece 10 can be provided.

  The power supply unit 25 is a power supply that supplies power to the control unit 23, the wireless communication unit 24, the storage unit 26, and the touch panel display 27. The power supply unit 25 can be wirelessly charged by the cradle as described above.

  Next, sound data processing by the core unit 20 will be further described. FIG. 4 is an explanatory diagram for explaining processing of sound data by the core unit 20. As shown in FIG. 4, the sound data collected by the microphone 21a and the microphone 21b is converted into a digital signal by the AD conversion unit 22a and the AD conversion unit 22b, and then output to the recording management unit 23d.

  The recording management unit 23d stores the sound data collected by the microphone 21a and the microphone 21b as recording data in the storage unit 26 and outputs the sound data to the auditory complement processing unit 23a. In addition, the recording management unit 23d can perform recording and reproduction by reading the recording data from the storage unit 26 and outputting it to the auditory complement processing unit 23a.

  The auditory complement processing unit 23a includes a direction identification unit 23a1, a personal identification unit 23a2, an emotion identification unit 23a3, a filter selection unit 23a4, an amplification unit 23a5, a voice filter 23a6, a life sound filter 23a7, a vehicle running sound filter 23a8, and an emergency vehicle sound filter. 23a9.

  The direction identification unit 23a1 identifies the arrival direction of sound using sound data acquired by the microphone 21a (output of the AD conversion unit 22a) and sound data acquired by the microphone 21b (output of the AD conversion unit 22b). Specifically, the direction identification unit 23a1 calculates a difference in sound arrival time in the sound data respectively acquired by the microphone 21a and the microphone 21b, and identifies a sound arrival direction based on the calculated difference in arrival time. The direction identification unit 23a1 generates sound direction data indicating the direction of arrival of the identified sound and uses it for display control of the display unit 27a.

  The personal identification unit 23a2 identifies a voice speaker using sound data acquired by the microphone 21a. Specifically, the personal identification unit 23a2 calculates voice feature parameters from the sound data acquired by the microphone 21a, and identifies voices having the same feature parameters as voices of the same person. Arbitrary methods, such as MFCC (Mel-Frequency Cepstrum Coefficient), can be used for calculation of the voice feature parameter.

  When the speaker is identified, the personal identification unit 23a2 uses information about the speaker as speaker identification data for display control of the display unit 27a. In addition, if a voice feature parameter and a personal name are associated with each other for a specific speaker and stored in the storage unit 26 as registered data, the feature parameter calculated from the sound data acquired by the microphone 21a is the feature parameter of the registered data. It is also possible to display the personal name of the registered data when it matches.

  The emotion identifying unit 23a3 identifies the speaker's emotion using the sound data acquired by the microphone 21a. Specifically, the emotion identification unit 23a3 detects changes in the volume, height, and pitch of the voice included in the sound data acquired by the microphone 21a, and identifies the speaker's emotion based on the detected changes. The voice when a speaker speaks with excitement tends to be louder, the range is higher, and the language is faster than the voice when speaking quietly. Therefore, it is possible to estimate the change in the speaker's emotion by detecting the change in the volume, height and pitch of the speech. The emotion identification unit 23a3 generates speaker emotion data indicating the emotion of the identified speaker and uses it for display control on the display unit 27a.

  The sound filter 23a6 is a filter that extracts a frequency band corresponding to sound from sound data (output of the AD conversion unit 22a) acquired by the microphone 21a and outputs the same as sound data. The life sound filter 23a7 is a filter that extracts the frequency band corresponding to the life sound from the sound data acquired by the microphone 21a and outputs it as the life sound data. The vehicle travel sound filter 23a8 is a filter that extracts a frequency band corresponding to the vehicle travel sound from the sound data acquired by the microphone 21a and outputs the same as the vehicle travel sound data. The emergency vehicle sound filter 23a9 is a filter that extracts the frequency band corresponding to the siren of the emergency vehicle from the sound data acquired by the microphone 21a and outputs it as emergency vehicle sound data. In addition, although the case where four types of filters were used was illustrated here, arbitrary filters can be added.

  The filter selection unit 23a4 functions as a situation determination unit that selects whether or not each filter can be used in accordance with the usage situation of the auditory complementation system. The usage status can be determined by a user's operation input, but the cradle ID of the cradle that was last used for charging is obtained from the charge management unit 23e, and can be determined according to the cradle ID.

  FIG. 5 is an explanatory diagram for explaining selection of a filter. FIG. 5A shows whether or not each filter can be used for the four usage situations of the scene type “home”, the scene type “driving”, the scene type “workplace”, and the scene type “recording / playback”.

  In the scene type “home”, the sound filter 23a6 is turned off (not used), the life sound filter 23a7 is turned on (used), the vehicle running sound filter 23a8 is turned off, and the emergency vehicle sound filter 23a9 is turned off. The reason why the sound filter 23a6 is off is because sound extraction is performed in the earpiece 10. Therefore, when using the hearing complementation system at home, the earpiece 10 outputs the voice and life sound with emphasis.

  In the scene type “Driving”, the sound filter 23a6 is off, the life sound filter 23a7 is off, the vehicle running sound filter 23a8 is on, and the emergency vehicle sound filter 23a9 is on. The reason why the sound filter 23a6 is off is because sound extraction is performed in the earpiece 10. Therefore, when using the auditory supplement system during driving, the earpiece 10 emphasizes and outputs voice, vehicle running sound, and emergency vehicle sound.

  In the scene type “workplace”, the sound filter 23a6 is off, the life sound filter 23a7 is off, the vehicle running sound filter 23a8 is off, and the emergency vehicle sound filter 23a9 is off. In this case, the output of the filter is not transmitted to the earpiece 10, but since the voice extraction is performed in the earpiece 10, the voice is emphasized and output.

  In the scene type “recording / playback”, the sound filter 23a6 is on, the life sound filter 23a7 is off, the vehicle running sound filter 23a8 is off, and the emergency vehicle sound filter 23a9 is off. At the time of recording / playback, voice extraction cannot be performed by the earpiece 10, and therefore voice extraction is performed by the core unit 20. When recording / reproducing is performed, the user may be able to individually select ON / OFF of each filter.

  FIG. 5B is a diagram showing the correspondence between the cradle ID and the scene type. In FIG. 5B, the cradle with the cradle ID “C001” is installed at home, the cradle with the cradle ID “C002” is installed in the vehicle, and the cradle with the cradle ID “C003” is installed. Is installed in the workplace. Therefore, if the cradle ID of the last cradle used for charging is “C001”, the scene type is determined to be “home”, and if the cradle ID of the last cradle used for charging is “C002”, the scene type is determined. If the cradle ID of the cradle last used for charging is “C003”, the scene type can be determined as “workplace”.

  When the recording management unit 23d outputs the recording data read from the storage unit 26 to the auditory complement processing unit 23a, the scene type “recording / playback” is used regardless of the cradle ID of the cradle used for the last charging. To be determined.

  Returning to FIG. 4, the description will be continued. The filter selection unit 23a4 outputs the sound data acquired by the microphone 21a to the filter set to “ON” according to the determined scene type. Each filter extracts a predetermined frequency band from the sound data acquired by the microphone 21a and outputs it to the amplifying unit 23a5.

  The amplifying unit 23a5 synthesizes and amplifies the outputs of the filters, and outputs them to the earpiece output processing unit 23c. That is, the sound data output from the amplifying unit 23a5 includes emphasized sound data such as sound data, life sound data, vehicle running sound data, and emergency vehicle sound data. The amplification factor for the output of each filter can be individually set according to the characteristics of the hearing ability of the user.

  The earpiece output processing unit 23 c transmits the sound data output from the hearing complement processing unit 23 a to the earpiece 10.

  Next, the identification of the sound arrival direction will be further described. FIG. 6 is an explanatory diagram for explaining identification of a sound arrival direction. Since the microphone 21a and the microphone 21b are arranged at positions separated from each other on the casing of the core unit 20, as shown in FIG. The direction identification unit 23a1 calculates a difference dT between the arrival time of the sound to the microphone 21a and the arrival time of the sound to the microphone 21b.

  As shown in FIG. 6B, if the difference dT is greater than 0, the direction of sound arrival is on the microphone 21a side, and if the difference dT is smaller than 0, the direction of sound arrival is on the microphone 21b side. Therefore, the direction of arrival of sound can be identified by the arrangement of the microphones 21a and 21b and the difference dT.

  Next, speaker emotion identification will be further described. FIG. 7 is an explanatory diagram for explaining identification of a speaker's emotion. As shown in FIG. 7, the emotion identification unit 23a3 detects changes in the size, height, and pitch of the voice included in the sound data acquired by the microphone 21a.

  In FIG. 7, the voice in the voice section R2 has a larger amplitude, a higher sound range, and a faster pitch than the voice in the voice section R1. For this reason, the emotion identification unit 23a3 estimates that the speaker's emotion in the speech segment R2 is more excited than the speaker's emotion in the speech segment R1. In this case, the emotion identification unit 23a3 controls display on the display unit 27a of an emotion display that schematically shows that the person is in an excited state.

  In addition, the voice in the voice section R3 has a smaller amplitude, lower sound range, and slower pitch than the voice in the voice section R1. For this reason, the emotion identification unit 23a3 estimates that the speaker's emotion in the voice segment R3 is more calm than the speaker's emotion in the voice segment R1. In this case, the emotion identification unit 23a3 controls display on the display unit 27a of an emotion display schematically showing that the emotion identification unit 23a3 is in a calm state.

  Next, filtering for sound data will be described. FIG. 8 is an explanatory diagram for explaining filtering on sound data. As shown in FIG. 8A, the sound filter 23a6 extracts sound data in a frequency band corresponding to sound from sound data output from the microphone 21a.

  Similarly, the life sound filter 23a7 extracts sound data in the frequency band corresponding to the life sound, the vehicle travel sound filter 23a8 extracts sound data in the frequency band corresponding to the vehicle travel sound, and the emergency vehicle sound filter 23a9 is the emergency sound. Sound data in the frequency band corresponding to the vehicle siren is extracted.

  Therefore, when the filter selection unit 23a4 selects the audio filter 23a6 and the life sound filter 23a7, as illustrated in FIG. 8B, the amplification unit 23a5 includes the sound data of the frequency band extracted by the audio filter 23a6, The sound data of the frequency band extracted by the life sound filter 23a7 is synthesized and amplified.

  Next, a procedure for processing sound data by the core unit 20 will be described. FIG. 9 is a flowchart showing a sound data processing procedure performed by the core unit 20. The core unit 20 acquires the outputs of the microphone 21a and the microphone 21b by converting them into digital signals using the AD conversion unit 22a and the AD conversion unit 22b (step S101). The recording management unit 23d stores the sound data collected by the microphone 21a and the microphone 21b in the storage unit 26 as recording data (step S102).

  The direction identifying unit 23a1 in the auditory complement processing unit 23a calculates the difference in sound arrival time from the outputs of the microphone 21a and the microphone 21b, and identifies the sound arrival direction based on the calculated difference in arrival time (step S103).

  The personal identification unit 23a2 in the auditory complement processing unit 23a calculates a voice feature parameter from the output of the microphone 21a, and identifies a voice having the same feature parameter as a voice of the same person (step S104).

  The emotion identifying unit 23a3 in the auditory complement processing unit 23a detects changes in the volume, height, and pitch of the voice included in the output of the microphone 21a, and identifies the speaker's emotion based on the detected changes (step S105). .

  The auditory complement processing unit 23a displays the sound arrival direction identified by the direction identification unit 23a1, the speaker identification result by the personal identification unit 23a2, and the speaker emotion identified by the emotion identification unit 23a3 on the display unit 27a. Control is performed (step S106).

  The filter selection unit 23a4 determines the scene type based on the user's operation input, the cradle ID of the cradle used for the last charging, whether or not the recorded data is being reproduced (step S107), and the determined scene type is set. A corresponding filter is selected (step S108).

  The filter selected by the filter selection unit 23a4 extracts a predetermined frequency band from the output of the microphone 21a and performs filtering to output as emphasized sound data (voice data, life sound data, vehicle running sound data, emergency vehicle sound data, etc.). This is performed (step S109). The amplifying unit 23a5 synthesizes and amplifies the emphasized sound data output from each filter (step S110). The earpiece output processing unit 23c transmits the sound data obtained as a result of the synthesis to the earpiece 10 (step S111), and ends the process.

  As described above, the auditory complementation system according to the first embodiment extracts the direction of sound arrival and the emotion of the speaker from the sound data collected by the microphone 21a and the microphone 21b of the core unit 20 as auditory supplement information. By controlling the display on the display unit 27a, it is possible to efficiently complement the user's hearing visually and provide a comfortable living environment for the user.

  In addition, since sound data in the corresponding frequency band can be extracted and amplified for living sounds, etc., and can be output from the earpiece 10, the user can also listen to sound data other than voice, so the user Can provide a comfortable living environment.

  In addition, it is possible to determine the surrounding situation when the hearing complementation system is used, extract and amplify sound data in a frequency band corresponding to the surrounding situation, and output it from the earpiece 10, so that the user can respond to the situation. Various sounds can be heard.

  The core unit 20 can be wirelessly charged with a cradle, and the earpiece 10 can be wirelessly charged with a pod, cradle, or core unit 20. Since the earpiece 10 can be wirelessly charged by the core unit 20, the earpiece 10 can be charged even in a situation where there is no pod or cradle on the go.

  In Example 1, although the structure which performs auditory complementation in the auditory complementation system which has the earpiece 10 and the core unit 20 was shown, the structure which performs auditory complementation alone in this Example 2 is demonstrated.

  FIG. 10 is a block diagram illustrating a configuration of an earpiece 110 that is a hearing complement apparatus according to the second embodiment. As illustrated in FIG. 10, the earpiece 110 includes a microphone 11, an AD conversion unit 12, a control unit 113, a power supply unit 15, a DA conversion unit 16, and an earphone 17.

  The microphone 11 is a sound collecting unit that collects sound data arriving at the user and outputs it as an analog signal. The AD conversion unit 12 converts the analog signal output from the microphone 11 into a digital signal and outputs the digital signal to the control unit 113.

  The control unit 113 is a DSP (Digital Signal Processor) that performs digital signal processing. The control unit 113 includes a filter selection unit 113a, a sound filter 113b1, a life sound filter 113b2, a vehicle running sound filter 113b3, an emergency vehicle sound filter 113b4, an amplification unit 113c, and a charge management unit 113d.

  The sound filter 113b1 is a filter that extracts a frequency band corresponding to sound from sound data acquired by the microphone 11 (output of the AD conversion unit 12) and outputs it as sound data. The living sound filter 113b2 is a filter that extracts the frequency band corresponding to the living sound from the sound data acquired by the microphone 11 and outputs the extracted frequency band as the living sound data. The vehicle travel sound filter 113b3 is a filter that extracts a frequency band corresponding to the vehicle travel sound from the sound data acquired by the microphone 11 and outputs the frequency band as vehicle travel sound data. The emergency vehicle sound filter 113b4 is a filter that extracts the frequency band corresponding to the siren of the emergency vehicle from the sound data acquired by the microphone 11 and outputs it as emergency vehicle sound data. In addition, although the case where four types of filters were used was illustrated here, arbitrary filters can be added.

  The charge management unit 113d is a processing unit that manages charging of the earpiece 110. The charge management unit 113d monitors the charge amount of the earpiece 110 and notifies the user as necessary. In addition, the charge management unit 113d identifies the device that charged the earpiece 110, and outputs a device ID that identifies the device to the filter selection unit 113a.

  The filter selection unit 113a selects whether or not each filter can be used in accordance with the usage status of the earpiece 110. The usage status can be determined by a user operation input, but can also be determined according to the device ID output by the charge management unit 113d.

  The filter selection unit 113a outputs the sound data acquired by the microphone 11 to the filter selected according to the determined usage situation. Each filter extracts a predetermined frequency band from the sound data acquired by the microphone 11 and outputs it to the amplifying unit 113c.

  The amplifying unit 113c synthesizes and amplifies the outputs of the filters and generates output sound data. The output sound data generated by the amplification unit 113c is converted into an analog signal by the DA conversion unit 16. The earphone 17 outputs a sound to the user's ear based on the analog signal output from the DA converter 16.

  The power supply unit 15 is a power supply that supplies power to the control unit 113, and the power supply unit 15 can be wirelessly charged by a pod, a cradle, or the like as in the first embodiment.

  Next, a procedure for processing sound data by the earpiece 110 will be described. FIG. 11 is a flowchart showing a sound data processing procedure by the earpiece 110. The earpiece 110 acquires the output of the microphone 11 by converting the output of the microphone 11 into a digital signal by the AD converter 12 (step S201).

  The filter selection unit 113a determines the usage status based on the user's operation input or the device ID of the device last used for charging (step S202), and selects a filter corresponding to the determined usage status (step S203). .

  The filter selected by the filter selection unit 113a performs filtering for extracting a predetermined frequency band from the output of the microphone 11 (step S204). The amplification unit 113c synthesizes and amplifies the filtering results of the filters, and generates output sound data (step S205). The earpiece 110 converts the output sound data, which is the synthesis result, into an analog signal by the DA converter 16 and outputs the analog signal from the earphone 17 (step S206), and ends the process.

  As described above, the earpiece 110, which is a hearing complementing apparatus according to the second embodiment, extracts and amplifies sound data in a frequency band corresponding to daily life from sound data collected by the microphone 11, and the earphone 17 Therefore, the user can hear sound data other than voice.

  In addition, it is possible to determine the surrounding situation when the hearing complementing device is used, extract and amplify sound data of a frequency band corresponding to the situation, and output it from the earphone 17, so that the user can respond according to the situation. Various sounds can be heard.

  In the first embodiment, the configuration in which the core unit 20 generates the auditory complement information from the sound data collected by the own device is illustrated. However, the sound data collected by the earpiece is transmitted to the core unit, and the core unit receives from the earpiece. Auditory supplement information may be generated from the sound data that has been processed. In the third embodiment, a configuration in which the core unit generates auditory supplement information using sound data received from the earpiece will be described.

  FIG. 12 is an explanatory diagram for explaining the hearing complementation system according to the third embodiment. As shown in FIG. 12, the hearing complementation system according to the third embodiment includes an earpiece 210 </ b> R, an earpiece 210 </ b> L, and a core unit 220.

  The earpiece 210R is a device worn on the user's right ear, and the earpiece 210L is a device worn on the user's left ear. Earpieces 210 </ b> R and 210 </ b> L each have a built-in microphone and earphone, and transmit sound data collected by the microphone to core unit 220.

  The auditory complement processing unit 223a in the core unit 220 is different from the auditory complement processing unit 23a shown in the first embodiment in that auditory supplement information is generated using sound data received from the earpieces 210R and 210L. The auditory complement processing unit 223a generates auditory supplement information including sound direction data and speaker emotion data using the sound data received from the earpieces 210R and 210L.

  The generation of the sound direction data and the speaker emotion data is the same as in the first embodiment, and the sound direction data and the speaker emotion data are displayed on the display unit 27 a of the core unit 220. Regarding the generation of sound direction data, the difference in sound arrival time in the sound data acquired by the earpiece 210R and the earpiece 210L is calculated, and the sound arrival direction is identified by the calculated difference in arrival time. .

  As described above, the core unit 220 can generate sound direction data, speaker emotion data, and the like and control the display on the display unit 27a as in the first embodiment. Although not shown, the core unit 220 can generate emphasized sound data such as life sound data and transmit it to the earpieces 210R and 210L as in the first embodiment. The earpieces 210R and 210L Emphasized sound data such as data can be output from the earphone.

  Next, the internal configuration of the earpiece 210R shown in FIG. 12 will be described. 13 is a block diagram showing an internal configuration of the earpiece 210R shown in FIG. As illustrated in FIG. 13, the earpiece 210 </ b> R includes a microphone 11, an AD conversion unit 12, a control unit 213, a wireless communication unit 214, a power supply unit 15, a DA conversion unit 16, and an earphone 17.

  The control unit 213 outputs the sound data collected by the microphone 11 and converted into a digital signal by the AD conversion unit 12 to the voice extraction unit 13 a and also to the wireless communication unit 214. The wireless communication unit 214 transmits the sound data output from the control unit 213 to the core unit 220, receives sound data such as life sound data from the core unit 220, and outputs the sound data to the control unit 213.

  Since other configurations and operations are the same as those of the earpiece 10 shown in the first embodiment, the same components are denoted by the same reference numerals, and description thereof is omitted. Further, the internal configuration of the earpiece 210L is the same as that of the earpiece 210R shown in FIG.

  Next, the internal configuration of the core unit 220 shown in FIG. 12 will be described. FIG. 14 is a block diagram showing an internal configuration of the core unit 220 shown in FIG. As illustrated in FIG. 14, the core unit 220 includes a control unit 223, a wireless communication unit 224, a power supply unit 25, a storage unit 26, and a touch panel display 27.

  The wireless communication unit 224 is a communication unit that performs wireless communication with the earpieces 210R and 210L and an external device. The wireless communication unit 224 outputs the sound data received from the earpieces 210R and 210L to the control unit 223, and transmits the sound data output from the control unit 223 to the earpieces 210R and 210L.

  The control unit 223 includes an auditory complement processing unit 223a, an earpiece output processing unit 23c, a recording management unit 23d, a charge management unit 23e, and a communication management unit 23f. The auditory complementation processing unit 223a is different from the first embodiment in that the auditory complementation processing unit 223a generates auditory complementation information that complements the user's hearing from the sound data collected by the earpieces 210R and 210L. Since other configurations and operations are the same as those of the core unit 20 shown in the first embodiment, the same components are denoted by the same reference numerals, and description thereof is omitted.

  As described above, the auditory complementation system according to the third embodiment extracts the arrival direction of the sound, the emotion of the speaker, and the like from the sound data collected by the earpieces 210R and 210L as auditory supplement information, and displays it on the display unit 27a. By controlling the display, the user's hearing can be efficiently supplemented visually, and a comfortable living environment can be provided to the user.

  In addition, since the sound data of the corresponding frequency band can be extracted and amplified for living sounds, etc., and output from the earpieces 210R and 210L, the user can also hear sound data other than voice, A comfortable living environment can be provided to the user.

  Moreover, since the surrounding situation when the hearing complementation system is used can be determined, sound data in a frequency band corresponding to the surrounding situation can be extracted and amplified, and output from the earpieces 210R and 210L, the user can It is possible to hear various sounds according to the sound.

  In addition, the structure and operation | movement shown to the said Examples 1, 2, and 3 do not limit this invention, It can implement by changing suitably. For example, in the first and third embodiments, the configuration in which the earpiece and the core unit communicate wirelessly is illustrated, but the earpiece and the core unit may be configured to communicate by wired connection.

  Moreover, in the said Example 1 and 3, although the structure which alert | reports the arrival direction of a sound and a speaker emotion by display was illustrated, these can also be alert | reported by the sound output from an earpiece. When the direction of sound arrival is notified by sound, a signal sound may be output from either the left or right earpiece. Also, when the speaker emotion is notified by sound, if the speaker emotion is identified as calm, the speech is output as it is, and if the speaker emotion is identified as being in an excited state or a calm state, the identification result It is only necessary to superimpose the signal sound matched to the sound and output it. As described above, the notification based on the sound arrival direction and the sound output of the speaker emotion can be used even in a configuration having no display unit as shown in the second embodiment.

  In the first and third embodiments, the earpiece is provided with a microphone, a voice extraction unit, and the like, and the earpiece alone can be used as a hearing aid. However, the earpiece may be configured to output only sound. . In this case, the sound filter of the core unit is always turned on, and the earpiece outputs the sound data received from the core unit as it is.

  As described above, the auditory complementation system, auditory complementation apparatus, and auditory complementation method according to the present invention are suitable for efficiently supplementing the user's hearing and providing a comfortable living environment for the user. Yes.

10, 110, 210R, 210L Earpiece 11, 21a, 21b Microphone 12, 22a, 22b AD conversion unit 13, 23, 113, 213 Control unit 13a Speech extraction unit 13b Output synthesis unit 14, 24, 214, 224 Wireless communication unit 15 , 25 Power supply unit 16 DA conversion unit 17 Earphone 20, 220 Core unit 23a, 223a Auditory supplement processing unit 23a1 Direction identification unit 23a2 Personal identification unit 23a3 Emotion identification unit 23a4, 113a Filter selection unit 23a5, 113c Amplification unit 23a6, 113b1 Audio filter 23a7, 113b2 Life sound filter 23a8, 113b3 Vehicle running sound filter 23a9, 113b4 Emergency vehicle sound filter 23c Earpiece output processing unit 23d Recording management unit 23e, 113d Charge management unit 23f Communication management unit 2 Storage unit 27 touch panel display 27a display portion 27b operating unit

Claims (9)

One or more sound collecting means for collecting sound data arriving at the user;
Voice extraction means for extracting voice from the sound data collected by the sound collection means;
From the sound data collected by the sound collecting means is used to complement the hearing of the user, the auditory complementary information extracting means for extracting the aural complementary information including at least living sound data,
Display control means for controlling display of the auditory complement information extracted by the auditory supplement information extracting means on a predetermined display means ;
A hearing complementing system comprising: sound output means for outputting to the user output sound data in which the life sound data is superimposed on the sound extracted by the sound extraction means. The aural complementary information extracting means, auditory completion system of claim 1 the sound of a predetermined frequency band from the sound data collected by the said sound collecting means, wherein the benzalkonium be extracted as the aural complementary information . A situation determination means for determining a surrounding situation of the user;
The aural complementary information extracting means, wherein the sound in a frequency band corresponding to the peripheral status from the sound data collected by the said sound collecting means to claim 2, wherein the benzalkonium be extracted as the aural complementary information Auditory complementation system. The auditory supplement information extracting means extracts the arrival direction of the sound included in the sound data collected by the sound collecting means as the auditory supplement information,
The auditory complementation system according to claim 1, 2 or 3, wherein the display control means controls the display means to display the direction of arrival of sound included in the sound data. A plurality of the sound collecting means;
5. The auditory sense according to claim 4, wherein the auditory complement information extracting unit determines an arrival direction according to a difference in arrival times of sounds in a plurality of sound data collected by the plurality of sound collecting units. Complementary system. The auditory supplement information extracting means extracts a speaker's emotion from the voice included in the sound data collected by the sound collecting means as the auditory supplement information,
The auditory complementation system according to any one of claims 1 to 5, wherein the display control unit controls display of the speaker's emotion on the display unit . The auditory complementation information extraction means detects a change in the size, height and / or pitch of the speech included in the sound data collected by the sound collection means, and the speaker's emotion estimated by the detected change Is extracted as the auditory supplement information. The auditory complement system according to claim 6. A hearing complement device worn on a user's ear,
One or more sound collecting means for collecting sound data arriving at the user;
Voice extraction means for extracting voice from the sound data collected by the sound collection means;
From the sound data collected by the sound collecting means is used to complement the hearing of the user, the auditory complementary information extracting means for extracting the aural complementary information including at least living sound data,
Display control means for controlling display of the auditory complement information extracted by the auditory supplement information extracting means on a predetermined display means ;
A hearing complementing device comprising: sound output means for outputting, to the user, output sound data obtained by superimposing the life sound data on the sound extracted by the sound extraction means. A sound collection step for collecting sound data arriving at the user;
A voice extraction step of extracting voice from the sound data collected by the sound collection step;
From the sound collecting step sound collected by the data, the used to supplement the user's hearing, the hearing complementary information extracting step of extracting the aural complementary information including at least living sound data,
A display control step for controlling display of the auditory supplement information extracted by the auditory supplement information extraction step on a predetermined display unit ;
And a sound output step of outputting to the user output sound data obtained by superimposing the life sound data on the sound extracted by the sound extraction step.

Images