JP7400364B2 - Speech recognition system and information processing method - Google Patents

Description

The present invention relates to a speech recognition system and an information processing method .

Smart speakers using voice recognition and multilingual translation systems are attracting attention. A smart speaker is a device that uses voice commands to operate indoor appliances and provide information such as weather forecasts. Multilingual translation systems are translation devices that use smartphones, dedicated terminals, etc. In a multilingual translation system, for example, a person's voice is input into a microphone that is a voice detection unit, the input voice is transcribed into text by voice recognition, then translated into a desired language by translation processing, and output from a speaker. Ru. Furthermore, systems that generate conversation records with customers at call centers and systems that automatically generate meeting minutes are being put into practical use, and these systems also utilize voice recognition technology.

Patent Document 1 discloses a technique for increasing the recognition rate of speech recognition by reducing speech recognition errors caused by noise other than human speech. The technology disclosed in Patent Document 1 uses a microphone to capture sounds generated during image capture by a camera, detects a speech section in which a person is speaking based on information in the image captured by the camera, and detects the speech section in which a person is speaking. It is configured to increase the sensitivity of human voice recognition in this section.

However, for example, in a situation where one microphone is installed at the center of a table and there are people around the table, the distance from the person's mouth to the microphone becomes relatively long. Therefore, unclear speech with a low S/N ratio is input, and informal utterances that deviate from grammar are frequently uttered. The conventional technology disclosed in Patent Document 1 does not assume speech recognition in such a situation, so there is room for improvement in increasing speech recognition accuracy.

In view of the above problems, the present invention can improve speech recognition accuracy even in situations where the distance from the mouth to the microphone is long.

In view of the above problems, a speech recognition system according to the present invention includes a speech acquisition device and a server, and the speech acquisition device includes a speech detection section that detects a plurality of speech sounds, and a speech detection section that detects a plurality of speech sounds. a synchronization control unit that performs control to synchronize audio data that is data indicating the content of the audio, and the server shares a teacher label with respect to the plurality of synchronized audio data and uses a speech recognition engine. Perform machine learning to recognize speech .

According to the present invention, it is possible to improve speech recognition accuracy even in situations where the distance from the mouth to the microphone is long.

A diagram showing a configuration example of a speech recognition system according to an embodiment of the present invention External view of audio acquisition device Hardware configuration diagram of audio acquisition device Functional block diagram of audio acquisition device Cloud server hardware configuration diagram Cloud server functional block diagram Diagram for explaining the operation of the speech recognizer, mechanical lip reader, and integrator Diagram to explain image features used for machine lip reading Diagram showing a first configuration example of a camera Diagram showing a second configuration example of the camera Flowchart to explain the operation of the speech recognition system External view of the housing with mute button Diagram showing an example of images before and after the mute button is pressed Diagram schematically showing how multiple microphones are arranged Diagram showing an example of audio data acquired by each of multiple microphones Diagram showing an example of a teacher label Diagram to explain the operation of the integrator

EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated using drawings. FIG. 1 is a diagram showing an example of the configuration of a speech recognition system according to an embodiment of the present invention. FIG. 1 shows a table 110 installed in a conference room 100, a plurality of people (conference attendees 31 to 36) existing around the table 110, and a voice recognition system 300.

The voice recognition system 300 improves voice recognition accuracy by acquiring the voice of one or more conference attendees 31 to 36 with a microphone and using voice data, which is data indicating the content of the acquired voice, for machine learning. It is configured as follows. In addition, the voice recognition system 300 improves voice recognition accuracy by capturing images of one or more conference attendees 31 to 36 with a camera and using captured data, which is data indicating the content of the captured images, for machine learning. It is configured as follows. Note that the speech recognition system 300 may be configured to collect only speech data to improve speech recognition accuracy. However, by collecting imaging data in addition to voice data, voice recognition accuracy can be further improved. Below, a configuration example will be described in which both voice data and image data are collected to improve voice recognition accuracy.

The voice recognition system 300 includes a voice acquisition device 1 installed in the center of a table 110, a whiteboard 120 installed between the wall of the conference room 100 and the table 110, and a cloud server 200. The voice data acquired by the voice acquisition device 1 is transmitted to the cloud server 200 via the whiteboard 120, and is subjected to voice recognition processing by a voice recognition engine or the like installed in the cloud server 200. The text data obtained as a result of voice recognition is sent to the whiteboard 120, where subtitles are displayed. Alternatively, the text data may be used to summarize the utterances as minutes. Note that the technology for automatically converting comments from meetings, lectures, interviews, etc. into text using voice recognition processing and the technology for creating minutes are well known as disclosed in Non-Patent Document 1, so a detailed explanation will be omitted. do.

The audio acquisition device 1 is a device that acquires the voices of a plurality of conference attendees 31 to 36 present around the table 110. Note that the audio acquisition device 1 is configured to acquire images of a plurality of conference attendees 31 to 36 in addition to audio. A configuration example of the audio acquisition device 1 will be described with reference to FIGS. 2A to 2C.

FIG. 2A is an external view of the audio acquisition device. FIG. 2A shows the appearance of the audio acquisition device 1 as well as the scenery of the conference room 100 captured by the audio acquisition device 1. The audio acquisition device 1 includes a housing section 2, a microphone 50 that is an audio detection section, and a camera 51 that is an imaging section. As the microphone 50, a multi-microphone capable of acquiring audio of multiple channels is used. A multi-camera capable of acquiring images of multiple channels is used as the camera 51. A multi-camera is, for example, a combination of a plurality of imaging units each having an angle of view of 90° or more.

The housing section 2 includes a disc-shaped pedestal section 1a installed on a table 110, and a columnar extension section 1b extending vertically from the pedestal section 1a and arranging a plurality of microphones 50 and the like at a position away from the table 110. Furthermore, the housing section 2 includes a disk-shaped unit installation section 1c that is provided on the upper part of the extension section 1b and in which a plurality of multi-microphones and multi-cameras are arranged. Note that the shape of the housing section 2 is not limited to the illustrated example, as long as it has a structure in which at least one camera 51 and microphone 50 can be provided.

One microphone 50 among the plurality of microphones 50 is provided at the upper part of the unit installation part 1c. The remaining microphones 50 are provided at a location other than the top of the unit installation portion 1c, for example, on a side surface of the unit installation portion 1c. The side surface portion is a portion of the entire outer peripheral portion of the unit installation portion 1c that includes, for example, a virtual plane parallel to a horizontal plane orthogonal to the vertical direction. A plurality of microphones 50 are installed on the side surface of the unit installation section 1c so as to be spaced apart from each other in the circumferential direction. By installing a plurality of microphones 50 in this manner, even when a plurality of conference attendees 31 to 36 are present surrounding the table 110, each microphone 50 can be placed to face each conference attendee. As a result, the distance from the microphone 50 to each conference attendee can be shortened, and clear audio with a high S/N ratio can be input.

FIG. 2B is a hardware configuration diagram of the audio acquisition device. The audio acquisition device 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an input device 104, a communication interface 105, and a bus 106.

The CPU 101 controls the entire audio acquisition device 1 by executing programs, and realizes each function described below. The ROM 102 stores various data including programs executed by the CPU 101. RAM 103 provides a work area for CPU 101. In addition to the microphone 50 and camera 51 described above, the input device includes a touch panel, a mouse, and the like for inputting information according to human operations. The communication interface 105 is an interface for connecting the audio acquisition device 1 to the communication network 301, for example, via a whiteboard 120, which is an example of an external device. The communication network 301 is a LAN (Local Area Network), the Internet, a mobile terminal network, or the like. The bus 106 interconnects the CPU 101, ROM 102, RAM 103, input device 104, and communication interface 105.

FIG. 2C is a functional block diagram of the audio acquisition device. The audio acquisition device 1 includes a start/end control section 10, a synchronization control section 11, a recording control section 12, a recording section 13, a mute control section 14, and a communication control section 15.

The start/end control unit 10 controls, for example, the start and end of recording by a plurality of microphones 50-1 to 50-n (n is an integer of 1 or more), and controls the start and end of recording by a plurality of cameras 51-1 to 51-n ( n is an integer of 1 or more) to control the start and end of imaging.

The synchronization control unit 11 performs control to synchronize a plurality of audio data acquired by one or more microphones 50-1 to 50-n, and synchronizes one or more captured data captured by one or more cameras 51. Performs control to synchronize. Details of the control by the synchronization control unit 11 will be described later.

The recording control unit 12 controls the recording of audio data and image data acquired by the microphone 50 and camera 51 into the recording unit 13 . The communication control unit 15 controls communication with external devices such as the whiteboard 120 and the cloud server 200. The communication control is, for example, control to transmit a plurality of audio data and image data controlled by the synchronization control unit to the cloud server 200 via the whiteboard 120 or directly to the cloud server 200.

Next, the configuration of the cloud server 200 will be described with reference to FIGS. 3A and 3B. FIG. 3A is a hardware configuration diagram of the cloud server. Cloud server 200 includes a processor 210, memory 220, and input/output interface 230.

The processor 210 is a calculation means composed of a microcomputer, a GPU (General Purpose Graphics Processing Unit), a system LSI (Large Scale Integration), and the like. The memory 220 is a storage means composed of RAM (Random Access Memory), ROM (Read Only Memory), and the like. The input/output interface 230 is an information input/output means for the processor 210 to input/output information to/from the audio acquisition device 1 . The processor 210, the memory 220, and the input/output interface 230 are connected to a bus 240, and can exchange information with each other via the bus 240. Bus 240 is connected to communication network 301 shown in FIG.

The cloud server 200 operates a virtual machine by, for example, having the processor 210 install virtual machine software (virtualization application) stored in the memory 220. Virtual machine software installs individual OSs by emulating individual hardware on a host OS (Operating System). This allows multiple virtual machines to run independently of each other on a single system. In this cloud environment, software for collecting data from the audio acquisition device 1 (data collection software), software for analyzing the data (analysis software), and the like are constructed. By using this virtualization technology, it is possible to efficiently utilize resources, reduce initial investment costs for hardware, and save power and space.

FIG. 3B is a functional block diagram of the cloud server. The cloud server 200 includes a speech recognition engine 201, a lip reading processing section 202, and an integrator 203.

The speech recognition engine 201 includes a speech feature extraction section 201a and a speech recognizer 201b. The lip reading processing unit 202 includes an image feature amount extraction unit 202a and a mechanical lip reading device 202b.

Next, the operations of the speech recognizer 201b, mechanical lip reader 202b, integrator 203, etc. will be explained with reference to FIGS. 4 and 5.

FIG. 4 is a diagram for explaining the operations of the speech recognizer, mechanical lip reader, and integrator. The audio feature extracting unit 201a extracts audio features, which are features used as input values for machine learning, from the audio data from the audio acquisition device 1. The audio feature extracting unit 201a inputs, for example, a plurality of audio data acquired by the audio acquisition device 1, cuts out each audio data for each unit time (every frame), and performs, for example, MFCC (Mel-Frequency). Cepstrum Coefficients: Calculates the spectral features of the audio signal for each frame, such as mel frequency cepstrum coefficients) and mel cepstrum features, and normalizes them.

The speech recognizer 201b performs machine learning using the feature extracted by the speech feature extraction unit 201a and recognizes speech. The speech recognizer 201b is a classifier that identifies feature amounts of speech, and a DNN (Deep Neural Network) can be exemplified as the classifier. A DNN has an input layer, an intermediate layer called a hidden layer, and an output layer. In DNN, a configuration is adopted in which the number of intermediate layers is increased to create a multilayer structure. In order to recognize speech using a DNN, it is most effective to perform supervised learning on the DNN using information called teacher labels or training data. Note that in order to realize a DNN, high computing power is required, so it is desirable to implement the DNN in the cloud server 200. It may be realized by the acquisition device 1. Note that in addition to DNN, the discriminator may use a method such as SVM (Support Vector Machine) or SIFT (Scale-Invariant Feature Transform).

There are various DNNs used for speech recognition, and one that has emerged recently is the End-to-End model. The End-to-End model is a method that processes input audio through a single neural network without dividing it into multiple functions such as an acoustic model, a language model, and a dictionary, as in the conventional method disclosed in Non-Patent Document 2. This is a model that converts directly into characters, and is also called a one-shot model. Although the End-to-End model has advantages such as simple structure, easy implementation, and fast response speed, it requires a large amount of training data.

The image feature extraction unit 202a extracts, for example, an image feature amount that is a feature amount as an input value for machine learning from the imaging data from the audio acquisition device 1. FIG. 5 shows an example of image feature amounts used in machine lip reading.

FIG. 5 is a diagram for explaining image feature amounts used in machine lip reading. First, the image feature extraction unit 202a recognizes, for example, the faces of conference attendees from the entire image captured by the camera 51. Facial recognition may use a general algorithm. Next, the image feature extraction unit 202a extracts lips from the recognized face. Then, the image feature extraction unit 202a extracts the time-series movement of each of the plurality of points plotted as shown in FIG. 5 as a feature from the extracted lip image. The feature amount is a feature amount of the mouths (lips) of the meeting participants imaged by the camera 51 in order to perform mechanical lip reading. The mechanical lip reader 202b performs machine learning using the feature amount. For example, in the case of a noisy meeting, the machine lip reader 202b performs machine learning using feature amounts of the mouths of each of the plurality of meeting attendees. Note that the method for extracting the feature amount is well known as disclosed in Non-Patent Document 3, so detailed explanation will be omitted.

The integrator 203 fuses the result of mechanical lip reading by the mechanical lip reader 202b with the result of voice recognition by the voice recognizer 201b. A method that uses not only the voice recognition result by the voice recognizer 201b but also a moving image of the lips during utterance is called multimodal voice recognition. In multimodal speech recognition, input moving images are converted into time-series image features, and the image features and audio features are fused to generate acoustic image features. Then, voice recognition is performed by using this acoustic image feature amount. Multimodal speech recognition is a valuable means of increasing speech recognition accuracy in meetings.

Next, a configuration example for improving recognition accuracy by machine lip reading will be described with reference to FIGS. 6A and 6B. FIG. 6A is a diagram showing a first configuration example of the camera. When the audio acquisition device 1 includes, for example, a camera 51 that is detachable from the housing 2, the camera 51 removed from the housing 2 may be installed on, for example, a whiteboard 120, as shown in FIG. 6A. I can do it. The installation method may be, for example, by providing a gripping means on the camera 51 and sandwiching the whiteboard 120 between the gripping means, or by providing a device with a fitting structure on each of the whiteboard 120 and the camera 51, and then holding them together. The camera 51 may be fixed to the whiteboard 120 by fitting together. With this configuration, the inside of the conference room 100 can be imaged from a location other than the table 110. As a result, even if the face direction of a meeting attendee changes, the mouth of the person can be imaged, increasing the probability that machine lip reading will be possible.

FIG. 6B is a diagram showing a second configuration example of the camera. In FIG. 6B, a camera 51-1, a camera 51-2, and a camera 51-2 forming a multi-camera are provided in the unit installation section 1c. In this case, each of the cameras 51-1, 51-2, and 51-2 captures images from different directions. Therefore, when a specific person speaks in a situation where there are multiple conference participants around the microphone 50, the voice is detected by the microphone 50, and an image of the person speaking is captured by the multi-camera. can do. Therefore, machine learning can be performed by combining the image of the person who makes the sound with the sound.

Note that the audio acquisition device 1 may be configured to have its height adjustable. For example, the unit installation part 1c of the audio acquisition device 1 is composed of two pipes with different diameters, and the inner pipe, which is the other thin pipe, is inserted inside the outer pipe, which is one thick pipe, and The outer tube is configured to be movable in the vertical direction. For example, when the area of the table 110 is small, the distance from the audio acquisition device 1 to the conference attendee tends to be short, so that the face and lips of the conference attendee may not fit within the viewing angle of the camera 51. In that case, by adjusting the height of the unit installation part 1c so that the face and lips of the meeting attendees are within the angle of view of the camera 51, it is possible to accurately capture the image of the person who is making the sound. , can be combined with voice for machine learning.

Next, the machine learning operation of the speech recognition system 300 will be described with reference to FIGS. 7 to 10. FIG. 7 is a flowchart for explaining the operation of the speech recognition system. FIG. 8A is an external view of a housing provided with a mute button. FIG. 8B is a diagram showing an example of images before and after the mute button is pressed. FIG. 9A is a diagram schematically showing a state in which a plurality of microphones are arranged. FIG. 9B is a diagram showing an example of audio data acquired by each of a plurality of microphones. FIG. 10 is a diagram showing an example of teacher labels.

When the audio acquisition device 1 is activated and the recording of the microphone 50 and the camera 51 are started (step S1), the recording and recording are continued until the mute button 20 shown in FIG. 8A is pressed (step S2, No. ).

The mute button 20 is, for example, a button for temporarily stopping recording of speech content including confidential information, or for erasing temporarily recorded speech content including confidential information going back a certain period of time. The mute button 20 may be used not only to temporarily stop recording or erase the uttered content going back a certain period of time, but also to temporarily stop recording or erasing recorded images going back a certain period of time.

The mute button 20 is provided, for example, in a casing connected to the audio acquisition device 1 via a cable, but may also be provided in the audio acquisition device 1. The mute button 20 may be of a shape that is easy for a person to operate or that makes it easy to identify whether recording is stopped or not, and may be of a push button type or a dial type. Here, an example of a push button type will be explained. In addition, an LED is provided next to the mute button 20, and the LED lights up while recording is in progress, and turns off while not recording, making it easy to understand the data acquisition status. Good too.

When the mute button 20 is pressed (step S2, Yes), audio recording is temporarily stopped (opt out) (step S3). For example, when the mute button 20 is pressed when a conference attendee starts speaking confidential information, the mute control section 14 generates a recording stop command and inputs it to the start/end control section 10 . The start/end control unit 10 that has received the recording stop command stops recording the confidential information by stopping the transmission of audio data from the microphone 50 to the recording control unit 12. As a result, highly confidential audio data is not recorded, and leakage of confidential information can be effectively prevented.

Note that the start/end control section 10 may stop transmitting the imaging data to the recording control section 12 together with the audio data when the recording stop command is input. With this configuration, even highly confidential imaging data is not recorded, making it possible to more effectively prevent leakage of confidential information.

The mute control unit 14 may be configured as follows. For example, when the mute button 20 is pressed after a conference attendee starts speaking confidential information, the mute control unit 14 controls the mute control unit 14 for a preset period of time (for example, from several seconds to several seconds) from the time the mute button 20 is pressed. An erasure command for erasing the recorded audio data up to the point in time (10 seconds) is generated and input to the recording control unit 12.

The recording control unit 12, which has received the erasure command, erases the audio data corresponding to the predetermined time period from among the audio data recorded in the recording unit 13 in chronological order. Furthermore, the mute control unit 14 generates a recording stop command and inputs it to the start/end control unit 10 at the same time as generating the erasure command, thereby stopping the transmission of audio data to the recording control unit 12. With this, for example, even if highly confidential audio data is temporarily recorded, the confidential information can be erased on the spot. Further, even if audio other than confidential information is recorded, if the recorded content is unnecessary for automatic minutes creation, etc., that part can be deleted, so the processing load on the cloud server 200 can be reduced.

Note that the recording control section 12 may erase not only the audio data but also the imaging data corresponding to the predetermined time from the recording control section 12 when the erasure command is input. With this configuration, even if highly confidential audio data and imaging data are temporarily recorded, the confidential information can be erased on the spot, and leakage of confidential information can be more effectively prevented. Furthermore, the resources of the recording section 13 can be used effectively. Furthermore, a large amount of voice data and imaging data for machine learning, which contribute most to improving the performance of the voice recognition engine 201, can be acquired while ensuring confidentiality.

Note that, when the mute button 20 is pressed, the mute control unit 14 controls, for example, as shown in FIG. 8B, to make the image during the meeting displayed on the display of the video conference system into a non-display state. may be configured. With this configuration, even if confidential information is being discussed, it is possible to prevent the contents from leaking to the outside. Note that when the mute button 20 is pressed again, audio recording is restarted, so that the image during the conference is displayed again on the display of the video conference system.

The mute control unit 14 may be configured to enable or disable the function of erasing part of the audio data and imaged data (step S4). For example, if the function is selected to be disabled (step S4, No), the process of step S6 is executed. When the function is selected to be enabled (step S4, Yes), the process of step S5, that is, data deletion (data erasure) is executed.

In step S6, the synchronization control section 11 performs control to synchronize the audio data detected by each of the plurality of audio detection sections. Note that the process of step S6 may be executed between step S1 and step S2. A synchronization control method in the synchronization control section 11 will be specifically described with reference to FIGS. 9A and 9B.

FIG. 9A is a diagram schematically showing a state in which a plurality of microphones are arranged. The symbols (1) to (6) shown in FIG. 9A are the first microphone (1), the second microphone (2), the third microphone (3), the fourth microphone (4), the fifth microphone (5), and Represents the sixth microphone (6). Each of these microphones has a different arrangement position and orientation. Furthermore, since each of these microphones is arranged around the table in the conference room, the distance from each microphone to the conference attendees around the table is relatively long.

FIG. 9B is a diagram showing an example of audio data acquired by each of a plurality of microphones. FIG. 9B shows the utterance content of a specific person detected by each of the second microphone (2), third microphone (3), and fourth microphone (4) among the plurality of microphones shown in FIG. 9A. The voice data represented is shown. These voice data represent the content of a specific person's utterances, but their waveforms are slightly different. The first cause is that the positions and orientations of the microphones are different. The second reason is that the distance from conference participants to each microphone is relatively long, so the voice emitted by a specific person may be reflected off the walls of the conference room 100 before reaching the microphone, or directly reach the microphone. This may result in differences in the reverberation of the audio to the microphone.

Therefore, for example, the sound pressure level of the feature point obtained by the second microphone (2) may be different from the sound pressure level of the feature point of the sound obtained by the third microphone (3).

Therefore, even if the voice is uttered by the same person, the waveform of the voice data detected by each microphone is slightly different, as shown in FIG. 9B. The synchronization control unit 11 thus synchronizes the timing of acquisition of a plurality of pieces of audio data having slightly different waveforms.

Furthermore, the synchronization control unit 11 performs similar processing between the third microphone (3) and the fourth microphone (4). As a result, the timings of specific feature points can be mutually matched between the microphones, and the timings at which the audio feature points are extracted can be input to the cloud server 200 at the same time. As a result, the accuracy of speech recognition can be efficiently improved.

Note that the synchronization control unit 11 may not only synchronize the sounds acquired by the plurality of microphones 50, but also synchronize the imaging by one or more cameras 51 in a similar manner. As a result, the teacher labels for machine learning in machine lip reading can be shared with those for voice recognition, and machine learning for voice recognition and machine lip reading can be advanced efficiently at low cost.

Next, referring to FIG. 10, teacher labels will be explained. As described above, when the positions and orientations of the plurality of microphones 50 are different, the waveform of the voice data and the feature amount of the voice corresponding to the content of speech by a specific person are different from each other. In this way, even if the waveform of the audio data or the feature amount of the audio differs, the content of the utterance is the same. Therefore, by performing machine learning (step S7) by sharing one teacher label as shown in FIG. The accuracy of speech recognition can be improved more efficiently than when machine learning is performed using teacher labels.

The teacher label is, for example, the utterance content (label) shown in FIG. 10, with the ``utterance number'' being ``0001'' and ``I have twisted all reality to my side.'' FIG. 10 also shows examples of a plurality of teacher labels. “Camera ID” is a number that identifies each of the plurality of cameras 51. "Speaker ID" is a number that individually identifies the person speaking. In addition, "gender ID", "start time" representing the time when the utterance started, "end time" representing the time when the utterance ended, etc. are associated with each other. The plurality of teaching labels shown in FIG. 10 are stored in the memory of the cloud server 200 in association with "utterance number", "camera ID", "speaker ID", etc. Note that the content of the teacher label is not limited to the illustrated example.

The teacher labels can be created by listening to the audio data and manually transcribing and time stamping it and using it for learning, or from the text output from the existing speech recognition engine 201 (speech recognition output) with confidence. There is a method of extracting output results with a high degree of accuracy as teacher labels. The former method of creating all teacher labels manually and performing machine learning is called supervised learning, and the latter method of using output results with high confidence as teacher labels without human intervention is semi-supervised learning. It is called. When performing semi-supervised learning in the speech recognition system 300 according to the present embodiment, if the recognition results based on speech data acquired by a plurality of microphones 50 are all the same, it is assumed that the confidence level is high and It can be considered to be used as a teacher label.

FIG. 11 is a diagram for explaining the operation of the integrator. The vertical axis in FIG. 11 is confidence level, and the horizontal axis is time. There are various methods of combining in the integrator 203, one example of which will be explained. Regarding the output of the speech recognizer 201b (for example, text information corresponding to speech), as shown in the sections indicated by symbols (1) and (2) in the figure, the confidence level is slightly lower than the threshold, or the confidence level is If the voice cannot be recognized because it is significantly lower than the threshold, the integrator 203 adopts the output of the mechanical lip reader 202b in these sections. On the other hand, when the confidence level of speech recognition is equal to or greater than the threshold value, the integrator 203 adopts the output of the speech recognizer 201b and does not adopt the output of the mechanical lip reader. This is because current machine lip reading is inherently less accurate than voice recognition.

As described above, the speech recognition device according to the present embodiment includes a speech detection section that detects a plurality of speech sounds, and a synchronization control section that performs control to synchronize the speech data that is data indicating the contents of the plurality of speech sounds. , and is configured to use a plurality of synchronized voice data for machine learning of a voice recognition engine. With this configuration, it is possible to synchronize and acquire voice data for machine learning, which contributes most to improving the performance of the voice recognition engine 201, even in meetings where voice recognition is difficult because the distance from the mouth to the microphone is long.

Note that beam forming, which detects a speaker and emphasizes the voice of the speaker, is known as a microphone array for voice recognition. Converting conference audio clearly using beamforming requires complicated signal processing, which makes voice recognition equipment very expensive, and it is also processed to differ from the original sound recording environment of the conference. Therefore, it is not possible to perform essential machine learning that is close to the original sound collection environment.

On the other hand, according to the speech recognition device according to the present embodiment, multiple pieces of speech data can be synchronized and used for machine learning without using beamforming, so complicated signal processing is not necessary. Therefore, it is possible to significantly improve speech recognition accuracy while suppressing an increase in the manufacturing cost of the speech acquisition device.

There is also a method of prioritizing high speech recognition accuracy in a conference and having multiple conference attendees each wear a headset, pin microphone, or the like. However, women in particular may dislike the unsanitary nature of using headsets and pin microphones over and over again.

On the other hand, according to the voice recognition device according to the present embodiment, high voice recognition accuracy can be ensured in a meeting without using a headset or the like, so the troublesomeness of wearing a headset or the like can be reduced. Furthermore, the user does not experience any discomfort caused by wearing a headset or the like.

Furthermore, since the conventional technology disclosed in Patent Document 1 uses a humanoid robot casing, it is difficult for meeting participants to concentrate on the meeting, and this can give a feeling of pressure, especially in a small meeting room.

On the other hand, according to the voice recognition device according to the present embodiment, as shown in FIG. 2A, it has a simple external shape resembling a tabletop lighting stand, so that it does not give a feeling of pressure to the conference participants. This will prevent your concentration from being hindered.

Further, the speech recognition device according to the present embodiment may be configured to include a recording unit that records a plurality of pieces of speech data. With this configuration, even if the cloud server is unable to receive voice data in real time due to communication failure, etc., machine learning using voice data can be continued by uploading the voice data stored in the recording unit to the cloud server. .

Further, the speech recognition device according to the present embodiment may be configured to include a communication control unit that communicates a plurality of speech data with an external device. With this configuration, multiple pieces of audio data can be sent to external devices such as whiteboards and cloud servers through the communication control unit, so voice data can be sent to external devices without having to install an expensive processor like a GPU in the speech recognition device. It is possible to realize machine learning using Therefore, even if the production volume of speech recognition devices increases, it is possible to suppress the increase in the cost of the entire system, and by performing machine learning using large amounts of data on external devices such as cloud servers, speech recognition accuracy can be improved. It can be significantly improved.

Furthermore, the speech recognition device according to this embodiment may be configured to include a mute control section that temporarily stops recording. Because much of what is said at a meeting includes highly confidential information, there may be times when it is not possible to record it, but by providing a mute control unit, recording can be stopped. Therefore, the user can participate in the conference without hesitating to speak, and as a result, a large amount of useful audio data can be collected. Therefore, training for informal utterances that deviate from grammar progresses, and speech recognition accuracy can be improved.

Further, the speech recognition device according to the present embodiment may be configured such that the plurality of speech detection sections are arranged in different positions, or the plurality of speech detection sections are arranged in different directions. With this configuration, audio from multiple channels can be acquired simultaneously, and even when multiple conference attendees are present around a table, each audio detection unit can be placed to face each conference attendee. Therefore, the distance from the audio detection unit to each conference attendee can be shortened, and clear audio with a high S/N ratio can be input.

Further, the speech recognition device according to the present embodiment may be configured to include an imaging section and use imaging data, which is data captured by the imaging section, for machine learning of a mechanical lip reader. With this configuration, the machine learning results of machine lip reading can be used as the machine learning results of the voice recognition engine, so it is possible to further improve the accuracy of voice recognition in meetings.

Further, the speech recognition device according to the present embodiment may be configured to include an integrator that adopts or does not adopt the results of machine learning of machine lip reading, depending on the results of machine learning of the speech recognition engine. With this configuration, if the voice is correctly recognized, the machine learning results of the voice recognition engine are prioritized, and if the voice is not recognized correctly, the output of the machine lip reader can be used, which allows for more accurate voice. Recognition can be achieved.

Further, in the information processing method according to the present embodiment, the speech recognition device performs control to synchronize the audio data, which is data indicating the contents of a plurality of voices acquired by the speech detection unit, and the server controls the synchronized plurality of voices. The voice data is used for machine learning of the voice recognition engine.

Further, the information processing program according to the present embodiment causes the speech recognition device to perform control to synchronize audio data, which is data indicating the contents of a plurality of sounds acquired by the audio detection unit, and causes the server to perform control to synchronize audio data that is data indicating the content of a plurality of sounds acquired by the audio detection unit. Perform machine learning on a speech recognition engine using multiple pieces of speech data.

1: Audio acquisition device 1a: Pedestal section 1b: Extension section 1c: Unit installation section 2: Housing section 10: Termination control section 11: Synchronization control section 12: Recording control section 13: Recording section 14: Mute control section 15: Communication Control unit 20: Mute button 31, 32, 33, 34, 35, 36: Conference attendees 50, 50-1, 50-n: Microphone 51, 51-1, 51-2, 50-n: Camera 100: Conference Room 101: CPU
102:ROM
103: RAM
104: Input device 105: Communication interface 106: Bus 110: Table 120: Whiteboard 200: Cloud server 201: Speech recognition engine 201a: Speech feature extraction unit 201b: Speech recognizer 202: Lip reading processing unit 202a: Image feature extraction Section 202b: Mechanical lip reader 203: Integrator 210: Processor 220: Memory 230: Input/output interface 240: Bus 300: Speech recognition system 301: Communication network

Patent No. 5797009

“Minutes preparation support system” [Retrieved October 9, 2021] Internet <URL: https://www.advanced-media.co.jp/products/service/private-enterprise-proceedings-preparation-support- system＞ “Basics of speech recognition” [Retrieved October 9, 2021] Internet <URL: https://www.slideshare.net/akinoriito549/ss-23821600> “Comparison of lip reading performance due to differences in facial regions used for recognition” [Retrieved October 9, 2021] Internet <URL: http://www.ii.is.kit.ac.jp/hai2011/proceedings/ pdf/II-2B-6.pdf＞

Claims (12)

A voice recognition system comprising a voice acquisition device and a server,
The audio acquisition device includes:
a voice detection unit that detects multiple voices;
a synchronization control unit that performs control to synchronize audio data that is data indicating the content of the plurality of audios;
Equipped with
The server is
The voice recognition engine performs machine learning on the multiple synchronized voice data using shared teacher labels to recognize voice.
Voice recognition system . The audio acquisition device includes:
The voice recognition system according to claim 1, further comprising a recording unit that records a plurality of said voice data. The audio acquisition device includes:
The voice recognition system according to claim 1 or 2, further comprising a communication control unit that communicates a plurality of the voice data with an external device. The audio acquisition device includes:
The voice recognition system according to any one of claims 1 to 3, further comprising a mute control unit that temporarily stops recording of the plurality of voices. 5. The voice recognition system according to claim 4, wherein the mute control unit erases the plurality of recorded voice data up to a certain point in time. The voice recognition system according to any one of claims 1 to 5, wherein the plurality of voice detection units have different arrangement positions or different orientations. The audio acquisition device includes:
Equipped with an imaging unit,
The voice recognition system according to any one of claims 1 to 6, wherein imaged data, which is data imaged by the imaging unit, is used for machine learning of machine lip reading. The server is
The speech recognition system according to claim 7, further comprising an integrator that adopts or does not adopt the machine learning result of the machine lip reading depending on the machine learning result of the speech recognition engine. The voice recognition system according to any one of claims 1 to 8, further comprising an imaging unit that is detachable from a housing of the voice acquisition device. The audio acquisition device includes:
Equipped with an imaging unit,
The synchronization control unit performs control to synchronize the audio data with imaging data that is data indicating the content of an image captured by the imaging unit,
The server is
Speech recognition according to any one of claims 1 to 9, wherein machine learning of a speech recognition engine is performed on a plurality of the synchronized audio data and the imaged data by sharing a teacher label to recognize speech. system . The audio acquisition device includes:
Any one of claims 1 to 10, comprising an imaging unit, and performs machine learning on the audio data and the imaging data that is data captured by the imaging unit using the same teacher label to recognize audio. The speech recognition system described in Section. An information processing method performed by a speech recognition system comprising a speech acquisition device and a server, the method comprising:
The voice acquisition device detects a plurality of voices with a voice detection unit,
The audio acquisition device performs control to synchronize audio data that is data indicating the content of a plurality of audios acquired by the audio detection unit,
An information processing method in which the server performs machine learning on a speech recognition engine by sharing teacher labels for a plurality of synchronized speech data to recognize speech .

Images