JP2019197550A - Sound input/output device - Google Patents

Abstract

To provide a sound input/output device for correctly recognizing sound in conformity with usage environment and reacting according to speaker identification and emotional state.SOLUTION: A sound input/output device according to one embodiment of the present invention includes: a sound receiving section having a plurality of microphones arranged three-dimensionally which can receive non-audible sounds as well as audible sounds; a sound generation section for generating audible sounds and/or non-audible sounds by one or a plurality of speakers; a signal processing section for processing and controlling a signal from the microphones; and a display section for performing a display on the basis of a processing result of the signal processing section; and a recording section for recording information on the sound collected by the sound receiving section.SELECTED DRAWING: Figure 1A

Description

  The present invention relates to a voice input / output device, for example, and more particularly to a voice input / output device suitable for a user's usage mode.

  In recent years, high performance of computers and communication devices has enabled advanced information processing via a network called a cloud in addition to high performance of terminal devices. In particular, a voice input / output device called an AI speaker having a voice input function for receiving voice input from a microphone (hereinafter abbreviated as “microphone”) and a voice output function for outputting voice from the speaker has become widespread. ing. In such a voice input / output device, it is required to correctly recognize the voice input from the microphone under various usage environments, to react by voice output or display without delay, and to correctly record the input voice. It is done.

  In this regard, Patent Document 1 discloses a technical idea of clearly recognizing a voice generated by a user in a usage environment in which a sound from a speaker, ambient noise, and a user's voice exist simultaneously. ing.

  Patent Document 2 discloses a technical idea that improves the accuracy of voice recognition when the user's voice and the voice output from the speaker overlap in time.

  However, Patent Documents 1 and 2 do not disclose a technical idea that is linked to more reliable speech recognition.

  In addition, neither of the above-mentioned documents describes in detail the audio recording. In particular, when speech is identified and recorded as a language, it becomes very difficult. The above-mentioned recording as a language is a language that is usually used by the user. For example, if the user is a Japanese, it means recording as Japanese type.

  On the other hand, Patent Literature 3 discloses a technical idea that, when a voice input / output device is operated, it reacts only when an activation word for activation is input from a microphone. The operation of the voice input / output device in this document is passive. In addition, since anyone can use the voice input / output device by entering a startup word (hot word), the user's hot word audio fingerprint is stored in advance, and the input hot word and A technique for ensuring security by starting only when a match is disclosed. However, it is difficult to determine a match / mismatch between the input hot word and the stored hot word audio fingerprint, and a more reliable security means is required.

JP 2001-94370 A Japanese Patent Laying-Open No. 2015-184530 JP 2017-76117 A

  This application pays attention to the conventional problems as described above, and can recognize the user's voice reliably even in an environment where there is a specific noise such as mechanical noise, laughter or alarm sound that exists in the environment of use. An object of the present invention is to provide an input / output device.

  It is another object of the present invention to provide a voice input / output device that embodies high-speed voice recognition processing technology for reducing user stress. It is another object of the present invention to provide a voice input / output device that actively searches for a use environment state and embodies the use of an optimum voice recognition technology. In the following description, the voice uttered by the user, the sound generated from the speaker, and the sound generated from the voice input / output surroundings of the present invention may be collectively referred to as a voice.

  In addition to the above, a voice input / output device that can also identify the user's identification, gender, and emotional state to improve voice recognition accuracy, and a voice input / output device that optimizes the response to user voice instructions Doing it is also an issue. It is another issue to provide a voice input / output device with more proactive talking and security measures.

  In order to solve the above-described problems, the aspect of the voice input / output device according to the present invention measures the usage environment using non-audible sound, performs an optimum process to match the measured environment, A positive man-machine interface device is identified by speaker identification and emotional state identification. Therefore, more specifically, the voice input / output device according to one aspect of the present application includes a voice reception unit in which a plurality of microphones that can receive audible to non-audible sounds are arranged three-dimensionally, and a single or a plurality of A sound generation unit that generates audible sound and / or non-audible sound through a speaker, a signal processing unit that controls processing of a signal from the microphone, a display unit that performs display based on a processing result of the signal processing unit, and the sound A voice input / output device having a recording unit for recording voice information collected by the reception unit can be provided.

  More specifically, the audio input / output device according to one aspect of the present application includes an audio receiving unit in which a plurality of microphones that can receive an audible sound to a non-audible sound are arranged in three dimensions, and an audible sound by one or a plurality of speakers. And / or a sound generation unit that generates a non-audible sound, a sound diffusion unit that diffuses sound generated by the sound generation unit, a signal processing unit that controls processing of a signal from the microphone, and a processing result of the signal processing unit A display unit that performs display based on the information, a recording unit that records audio information collected by the audio reception unit, an interface unit that performs information exchange with an external device by wire, and wirelessly exchanges information A power supply unit that supplies power to each unit of the communication unit, the sound reception unit, the sound generation unit, the sound diffusion unit, the signal processing unit, the display unit, the recording unit, the interface unit, and the communication unit; each It may be configured to include a housing that houses the.

  In the above, the audible sound is generally 20 Hz to 20 KHz, and the non-audible sound is a sound of other frequencies. As a non-audible sound for searching the surrounding environment of a voice input / output device to be described later, it is desirable to use a so-called ultrasonic wave in the vicinity of 30 KHz from the viewpoint of ease of generation and sound collection and resolution.

  In addition to the structure in the said aspect, this application is good also as an aspect which has a display part further comprised from a several light emission display and / or an image display. In this case, it is possible to change the display method of the light emission display or the image display device according to the surrounding environmental sound, speaker identification, or speaker emotion identification result.

  In the above aspect, the non-audible sound is intermittently generated, the reflected sound from the periphery of the device is received by the plurality of microphones, and the voice arrival information for grasping the environment around the device with respect to the two-dimensional direction and distance is grasped. A voice arrival information grasping function may be provided.

  Moreover, in the said aspect, you may make it further have an environmental sound identification function which can acquire the environmental sound identification information which is the information for identifying an environmental sound.

  Moreover, in the said aspect, you may make it further have a speaker identification function which can acquire the speaker identification information which is the information for identifying a speaker.

  Moreover, in the said aspect, you may make it further have a speaker emotion identification function which can acquire the speaker emotion information which is the information for identifying a speaker's emotional state.

  Moreover, in the said aspect, the speaker identification function which can acquire the speaker identification information which is information for identifying a speaker, and the speaker emotion which is information for identifying a speaker's emotional state A speaker emotion identification function capable of acquiring information, and when recording sound information input from the microphone in the recording unit, voice arrival information and speaker identification associated with the sound information Any one or more of information, speaker emotion information, and external information may be recorded substantially simultaneously.

  Further, in the above aspect, a light emission interval, a light emission color of the plurality of light emitting display units based on at least one of the sound arrival information, the speaker identification information, the speaker emotion information, and external information, Any one or more of the light emission orders may be changed.

  Moreover, in the said aspect, you may make it further have the mechanism and vibration mechanism which rotate the whole apparatus.

  Moreover, in the said aspect, you may make it further provide an imaging part.

  Moreover, in the said aspect, you may make it further provide a personal authentication part.

  Moreover, in the said aspect, you may make it further provide a projector part.

  Moreover, in the said aspect, you may make it further provide an infrared communication part.

  In addition to the configuration in the above aspect, the present application further detects intruders by generating non-audible sound and monitoring by a TV camera in addition to passive activation by activation words, and the voice input / output device itself is activated actively. It is also possible to adopt a mode further including identification functions such as transmission / reception of secret words, face recognition by a TV camera, fingerprint verification, and the like. In this case, in addition to the speaker identification described above, personal identification can be performed more reliably and security can be ensured.

  In the technical idea of the present application, for example, to improve customer satisfaction, it records what kind of emotion the speaker has about what kind of utterance, and uses the voice input / output device on the client side for the call center. This includes alerting the operator and reporting to the administrator. Also, when using the client's voice input / output device for a meeting, if the attendees become emotional, take a break to calm down, or speak a voice to the effect of calmness It is also included.

  In general, according to the present application, the optimum speech recognition technology that actively searches the use environment and matches the search result is used, or the environment sound existing in the use environment is recognized and the noise from the noise source existing in a specific direction is detected. It is possible to block input or identify the voice characteristics of the user. In addition, when recording the voices of a plurality of speakers, it is possible to identify which speaker is the voice recording. Further, for example, it is possible to automatically determine that the owner has returned home, and to perform an active operation such as saying “Please come home!”.

  By using a plurality of microphones, the two-dimensional direction of the speaker can be known by the beam forming technique, and the speaker's words can be reliably identified by separating from the ambient noise. If the main orientation identification information, the speaker identification information, the emotion identification information, and the external information are recorded together with the voice reception information, it is very useful for rearranging the voice information.

  When recording voice information converted into a language, it is very important to identify who generated the voice. However, in the case of recording just converted into language, the direction identification information is as described above. If speaker identification information, emotion identification information, and external information are recorded, reliable speaker identification is possible.

  As mentioned above, non-audible sound is intermittently generated in pulses, and the reflected sound is received by the above-mentioned multiple microphones, so it is possible to check the sound environment like the surrounding reflector, and the influence of multipath on sound wave propagation And the accuracy of voice identification can be further improved. Furthermore, when the reflector around the voice input / output device moves over time, it is determined that there is an intruder, and an unprecedented active function such as “welcome” or “return home” can be achieved. It becomes possible.

  In addition, by performing voice feature analysis such as frequency analysis, it is possible to know the speaker's identification and the speaker's emotional state, and display the display section appropriately, for example, to suppress the excitement state based on the result. be able to. This is a very useful effect for man-machine interface.

  Further, according to the present application, for example, what kind of emotion a speaker has for what kind of utterance is recorded, and if the voice input / output device on the client side is used for a call center, the operator is alerted. By reporting to the administrator, customer satisfaction can be improved. Also, when using the client's voice input / output device for a meeting, if the attendees become emotional, take a break to calm down, or speak a voice to the effect of calmness Through this, it is possible to provide an audio environment suitable for the situation and atmosphere.

  In addition to active activation by activation words, intruders are detected by generation of non-audible sound and monitoring by a TV camera, and the voice input / output device itself is activated actively, sending and receiving personal identification passwords, and TV Face recognition by the camera, fingerprint verification, and the like have the effect of ensuring personal security by ensuring personal identification in addition to speaker identification.

1 is a perspective view of a voice input / output device according to an embodiment of the present invention. It is a perspective view of the voice input / output device concerning another embodiment of the present invention. 1 is a schematic diagram of an internal structure of a voice input / output device according to an embodiment of the present invention. FIG. 4 is a perspective view for conceptually explaining the operation of the projector mounted on the voice input / output device according to the embodiment of the invention. It is a conceptual perspective view which shows an example of microphone arrangement | positioning of the audio | voice input / output device which concerns on one Embodiment of this invention. It is a conceptual perspective view which shows another example of microphone arrangement | positioning of the audio | voice input / output device which concerns on one Embodiment of this invention. It is an example of a block diagram of a voice input / output device concerning an embodiment concerning one embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following, the range necessary for the description for achieving the object of the present invention is schematically shown, and the range necessary for the description of the relevant part of the present invention will be mainly described. According to a known technique.

  1A and 1B are diagrams showing two embodiments of a voice input / output device according to an embodiment of the present invention. In FIG. 1A, all the electric circuits and the like to be described later are incorporated in a cylindrical casing 10 having an exterior material 12 made of punching metal or the like through which sound can freely enter and exit, and a light-emitting display section such as a multicolor LED at the top. The example summarized in the simple design which attached | subjected 11 is shown. The exterior material 12 is not limited to the above-described material, and any material can be applied as long as sound can freely enter and exit. The shape of the housing 10 is not limited to a cylindrical shape, and is rectangular or polygonal. Various shapes such as shapes are conceivable, all of which are included in the technical idea of the present application.

  FIG. 1B is an example in which the image display unit 13 is further incorporated in the form shown in FIG. 1A, the light emitting display unit 15 is incorporated at the top, and the housing base 16 is rotatable. A rotation mechanism 31 (described later) using a motor or the like is incorporated in the chassis base 16 so that the entire chassis can be rotated. Therefore, an imaging unit 33 such as a TV camera or an image display unit 24 (FIG. 2A) is directed toward the speaker. See). Furthermore, the entire casing can be vibrated using a motor used for the rotating mechanism, and can be used for an acknowledgment of voice input, emphasis of generated voice, and the like.

  Similarly, an infrared human sensor, a fingerprint sensor, and a TV camera may be installed at or near the top. In FIG. 1A and FIG. 1B, individual multicolor LEDs are continuously arranged in a circle, but various variations such as a square shape or a heart shape can be considered, and each individual LED corresponding to each variation can be considered. It is conceivable to employ a lighting interval, a lighting color, and a lighting sequence. In addition, the lighting sequence may be variously considered to indicate a voice arrival method or to be a speaker's emotion or a speaker's identification position, all of which are included in the technical idea of the present application.

  2A is an example of the internal structure of the voice input / output device shown in FIG. 1B according to an embodiment of the present invention. FIG. 2B is a diagram of a projector mounted on the voice input / output device according to an embodiment of the present invention. It is a perspective view for demonstrating an effect | action conceptually. As shown in FIG. 2A, a light emitting display unit 21 is arranged at the top of a cylindrical casing 20, and a plurality of microphone units 22 having a plurality of microphones 220 arranged at substantially equal intervals near the top, and a lower part thereof. Similarly, a microphone unit 23 in which a plurality of microphones 230 are arranged at substantially equal intervals is arranged. Between the microphone unit 22 and the microphone unit 23, an electric circuit such as an image display unit 24 and a signal processing unit described later is accommodated.

  2C is a conceptual perspective view showing an example of a microphone arrangement of the voice input / output device according to the embodiment of the present invention, and FIG. 2D is a conceptual perspective view showing another example of the microphone arrangement. In FIG. 2C, in addition to a microphone unit in which a plurality of microphones are arranged at equal intervals on a horizontal plane, similar microphone units are three-dimensionally separated on the vertical axis to thereby determine the two-dimensional arrival direction of the incoming sound source to each microphone. It can be measured. The arrangement position of the microphone is not limited to the arrangement example of FIG. 2C, but various arrangement methods such as, for example, arrangement at the position corresponding to the angular position of the polygonal column inscribed in the cylindrical housing as shown in FIG. 2D can be considered. All of these are included in the technical idea of the present application.

  Similarly, in FIG. 2A, a plurality of speakers are coaxially arranged downward and a substantially conical cone-shaped sound diffusion unit 30 is arranged in the lower part of the coaxial so that the sound generated from the plurality of speakers 25 and 26 is isotropically surrounded. Has spread. Of course, the plurality of speakers 25 and 26 and the sound diffusing unit 30 may be arranged upside down, and some other variations of the arrangement may be considered, but all of them are included in the technical idea of the present application. .

  In the form shown in FIGS. 1B and 2A, the image display unit 13 can be directed in the direction of the speaker by the above configuration, and a more effective man-machine interface can be obtained. In the form shown in FIG. 1A, with the same configuration (not shown), the direction of the speaker can be displayed by a plurality of microphones by the display of the light emitting display unit according to the orientation of the speaker.

  As will be described later, in addition to detection of an intruder by reflection of non-audible sound, a human sensor using infrared rays may be provided on the top of the housing 10 or the like. Similarly, a fingerprint sensor for ensuring personal identification or an imaging device such as a TV camera may be installed on the top. Furthermore, as shown in FIG. 2B, by providing the projector 34, it is possible to project an explanatory diagram and related images in an enlarged manner in synchronization with the audio output. As a scene to which this can be applied, for example, a mode in which a projector 34 according to the present embodiment projects a map or agenda on an indoor whiteboard, a wall, or a screen for explanation of a meeting or travel.

  FIG. 3 is a block diagram of the voice input / output device shown in FIG. 1B according to an embodiment of the present invention. As shown in the figure, the microphone unit 40 arranged on the upper and lower horizontal planes of the cylindrical case has an AGC (Automatic Gain Control “automatic gain control”) that adjusts the output level to the target value even if the input level of the system changes. It is input to a signal processing unit 42 mainly composed of a μCPU through a microphone control unit 41 that performs forming and the like. Further, the microphone control unit 41 can perform noise removal and echo cancellation via the interface unit 50.

  The signal processing unit 42 performs preprocessing for improving the identification accuracy such as ambient noise removal on the sound signal from the microphone. While extracting the arrival direction information of the processed audio signal, the information is transmitted to the outside from the communication unit 43 or the interface unit 50, and advanced information processing such as speaker identification processing and emotion identification processing is performed by cloud processing or the like. It records in the recording part 47 as audio | voice information with information. At the same time, a display suitable for the result of the information processing can be displayed on the display unit 46.

  Furthermore, in the signal processing unit 42, it is possible to record only the audio information from the specific azimuth without capturing the audio information from the noise source existing in the specific azimuth by the voice arrival azimuth information.

  In addition, the recording unit 47 has a multi-layer configuration, and records the recorded voice by linking related information such as the direction of arrival of voice information to be recorded, speaker identification, emotion identification, etc. and recording it on a layer different from the voice information. Organizing information becomes very easy.

  The signal processing unit 42 includes a communication unit 43 for wireless communication with the outside by Wi-Fi, Bluetooth (registered trademark), and the like, and an interface unit 50 for connecting to an external device by hard wires. For this reason, it is possible to receive ambient noise with an external microphone and input the received noise from the expansion port to reduce the influence of ambient noise, or to communicate with an external device via the USB port.

  Furthermore, instead of changing the TV channel or turning on / off the lighting device via the Internet by voice command, the infrared communication unit (IR transceiver unit) 35 is installed, so that the TV or TV can be directly controlled by voice command. It becomes possible to directly control the lighting device control and external equipment.

  According to the present invention, it is possible not only to correctly recognize an input voice signal but also to actively recognize the surrounding environment. Therefore, a push-type man who can actively talk to a speaker from the voice input / output device of the present invention. -It is expected to be widely used as a machine interface in home appliances, entertainment, and various industrial fields.

DESCRIPTION OF SYMBOLS 10 ... Case, 11 ... Light emission display part, 12 ... Exterior material, 13 ... Image display part, 14 ... Rotation part, 15 ... Light emission display part, 16 ... Case base, 20 ... Case, 21 ... Light emission display part, 22 ... Microphone unit, 23 ... Microphone unit, 24 ... Image display unit, 25 ... Speaker (audible sound generating unit), 26 ... Speaker (non-audible sound generating unit), 27 ... Audible sound, 28 ... Non-audible sound, 29 ... Base: 30 ... Audio diffusion unit, 31 ... Rotation mechanism, 32 ... Personal authentication unit, 33 ... Imaging unit, 34 ... Projector, 35 ... Infrared communication unit, 40 ... Microphone unit, 41 ... Microphone control unit, 42 ... Signal processing unit 43 ... Communication unit 44 ... Sound generation unit 45 ... Non-audible sound generation unit 46 ... Display unit 47 ... Recording unit 48 ... Rotation drive unit 49 ... Power supply unit 50 ... Interface unit

Claims (13)

A sound reception unit in which a plurality of microphones capable of receiving audible to non-audible sounds are arranged in three dimensions;
A sound generator that produces audible and / or non-audible sounds with one or more speakers;
A signal processing unit for processing and controlling the signal from the microphone based on an external command;
An interface unit for exchanging information with an external device by wire;
A wireless unit that wirelessly exchanges information with external devices;
A display unit for performing display based on a processing result of the signal processing unit;
A voice input / output apparatus comprising: a recording unit that records sound information collected by the sound receiving unit.   The voice input / output device according to claim 1, wherein the display unit includes a plurality of individual light emitters and / or image displays.   Voice arrival information grasping function for intermittently sounding the non-audible sound, receiving reflected sound from the periphery of the apparatus with the plurality of microphones, and grasping voice arrival information for grasping the environment around the apparatus with respect to the two-dimensional direction and distance The voice input / output device according to claim 1 or 2, characterized by comprising:   The voice input / output device according to any one of claims 1 to 3, further comprising a sound environment identification function for acquiring environmental information by sound.   The voice input / output device according to claim 1, further comprising a speaker identification function for acquiring speaker identification information.   The voice input / output device according to claim 1, further comprising a speaker emotion identification function for acquiring speaker emotion information. A speaker identification function capable of acquiring speaker identification information, which is information for identifying a speaker,
A speaker emotion identification function capable of obtaining speaker emotion information, which is information for identifying a speaker's emotion state, and
When recording the sound information input from the microphone in the recording unit, at least one of sound arrival information, speaker identification information, speaker emotion information, and external information associated with the sound information is substantially simultaneously. 4. The voice input / output device according to claim 3, wherein recording is performed.   Based on at least one of the sound arrival information, the speaker identification information, the speaker emotion information, and the external information, any one of a light emission interval, a light emission color, and a light emission order of the plurality of light emission display units. 8. The voice input / output device according to claim 7, wherein one or more can be changed.   9. The voice input / output device according to claim 1, further comprising a mechanism for rotating the entire device and a vibration mechanism.   The voice input / output device according to claim 1, further comprising an imaging unit.   The voice input / output device according to claim 1, further comprising a personal authentication unit.   The voice input / output device according to claim 1, further comprising a projector unit. The voice input / output device according to claim 1, further comprising an infrared communication unit.

Images