JP5849761B2 - Speech recognition system, speech recognition method, and speech recognition program - Google Patents

Description

  The present invention relates to a voice recognition system using image recognition technology.

  Speech recognition is a promising technology and is beginning to be used in smart devices. By using the voice recognition technology, it is possible to more easily perform operations that have been performed manually until now, such as creating notes, searching, or operating the system.

  Various measures have been taken to improve the recognition accuracy of speech recognition, such as performing noise processing and preparing a speech dictionary, but it is difficult to achieve 100% recognition accuracy. However, customers who use speech recognition expect almost 100% recognition accuracy and are not able to meet the needs at present. Therefore, research is still being conducted to develop technologies and solutions for improving recognition accuracy.

  A general configuration for performing speech recognition is roughly divided into a microphone (hereinafter referred to as a microphone) that collects speech and a recognition engine portion that analyzes the obtained data and converts it into text. The accuracy of recognizing speech greatly depends on the quality of the collected speech data and the speech dictionary used in the recognition engine. When attention is focused on voice collection here, for example, there is a problem that noise around the recognition target person or speech other than the recognition target person is collected by a microphone, thereby lowering the quality of the voice data and lowering the recognition accuracy. .

  As a technique for solving such a problem, for example, in Patent Document 1, a directional microphone is used to enhance the sound collecting ability by emphasizing and inputting the sound generated by the recognition target person, and the subject in the captured image Describes a technique for controlling the directivity of the voice input unit based on the range occupied by the voice input unit. Further, in Patent Document 1, it is determined that there is a high possibility that a person whose face is recognized as having an open mouth is speaking, and the input voice is emphasized from the range occupied by the person's face. A technique for inputting voice with gender is described.

JP 2011-61461 A

  However, since the method of moving the mouth when speaking is characterized by individuals, even if the technique described in Patent Document 1 is used, there is a case where the speaker who is the recognition target cannot be reliably identified. is there. For this reason, voice collection for a speaker is not successful, ambient noise and the like are collected, the quality of voice data is lowered, and voice recognition accuracy may be lowered.

  Accordingly, an object of the present invention is to provide a speech recognition system that can improve the accuracy of speech recognition.

  The speech recognition system according to the present invention obtains an image of a user who is a target of speech recognition from a camera, uses the image to identify the user, and a pre-stored personal mouth. A mouth movement database storing the movement feature amount of the mouth, detecting the mouth state of the user from the image, and storing the mouth movement feature amount corresponding to the user stored in the mouth movement database And a mouth movement determination means for determining whether or not the user is speaking, and determining that the user is speaking A direction-of-direction determining means for notifying a voice input means for acquiring the voice of the user, and a voice recognition means for acquiring the voice and performing voice recognition. And

  According to the speech recognition method of the present invention, an image obtained by capturing a user who is a target of speech recognition is acquired from a camera, the user is identified using the image, and mouth movement characteristics for each individual stored in advance are stored. A mouth movement database storing the amount, detecting a mouth state of the user from the image, and determining the mouth movement feature amount corresponding to the user stored in the mouth movement database and the image Compare the obtained feature value of the mouth movement of the user, determine whether the user is speaking, and if it is determined that the user is speaking, obtain the voice of the user The position of the user is notified to a voice input means for acquiring the voice, and voice recognition is performed.

  The speech recognition program according to the present invention obtains, from a camera, an image obtained by capturing a user who is a target of speech recognition from a camera, a face recognition process for identifying the user using the image, and a personally stored person. A mouth movement database that stores a feature value of each mouth movement, detects a mouth state of the user from the image, and moves the mouth corresponding to the user stored in the mouth movement database; The mouth movement determination process for comparing whether the user is speaking by comparing the feature amount of the user and the feature amount of the user's mouth movement obtained from the image, and the user speaking If it is determined, a direction determination process for notifying the user's position to voice input means for acquiring the user's voice and a voice recognition process for acquiring the voice and performing voice recognition are executed. That features To.

  According to the present invention, the accuracy of voice recognition can be improved.

It is a block diagram which shows the structure of embodiment of the speech recognition system by this invention. It is a sequence diagram which shows operation | movement of embodiment of the speech recognition system by this invention. It is a flowchart which shows operation | movement of embodiment of the speech recognition system by this invention. It is explanatory drawing which shows operation | movement of the determination function of the movement of the mouth with respect to a specific person. It is explanatory drawing which shows operation | movement of the speech recognition function with respect to a specific person. It is explanatory drawing which shows operation | movement of a sentence end determination function. It is a block diagram which shows the principal part of the speech recognition system by this invention.

  FIG. 1 is a block diagram showing a configuration of an embodiment of a speech recognition system according to the present invention. As shown in FIG. 1, the speech recognition system of the present embodiment includes a face recognition function (face recognition unit) 30, a mouth movement determination function (mouth movement determination unit) 40, a voice data input presence / absence determination function (voice data input). Presence / absence determination unit) 50, directivity direction determination function (directivity direction determination unit) 60, speech recognition function (speech recognition unit) 70, and sentence end determination function (sentence end determination unit) 80 are provided. Each of the above functions other than the sentence end determination function 80 is connected to the device 10 used by the user via the network 20 and transmits and receives data. The device 10 includes a camera 11 and a directional microphone.

  The face recognition function 30 includes a feature amount extraction unit 31, a DB (Data Base: database) matching unit 32, and a face DB (Data Base: database) 33. The face recognition function 30 inputs an image obtained from the camera 11. The feature amount extraction unit 31 extracts the feature amount of the input image. The DB collation unit 32 collates the feature quantity with the face feature quantity stored in advance in the face DB 33 to identify the user.

  Mouth movement determination function 40 includes mouth state detection means 41, specific user DB (Data Base) setting means 42, individual mouth movement DB (Data Base: database) 43, and mouth movement during speech. The determination means 44 is included. The mouth state detecting means 41 detects the mouth state from the image obtained from the camera 11. The mouth movement determination unit 44 at the time of utterance compares the data of the individual mouth movement DB 43 set by the DB setting unit 42 of the specific user with the image obtained from the camera 11.

  The voice data input presence / absence determination function 50 includes a voice data acquisition unit 51 and a voice data presence / absence determination unit 52. The audio data acquisition unit 51 acquires audio data from the device 10. The sound data presence / absence determining means 52 determines the presence / absence of sound data. For example, the sound data presence / absence determination unit 52 determines that sound data is included if the volume of the acquired sound data exceeds a predetermined value.

  The orientation direction determination function 60 includes a search result acquisition unit 61, a target person image position determination unit 62, and a direction determination unit 63. The search result acquisition unit 61 acquires the results output by the face recognition function 30 and the mouth movement determination function 40. The target person image position determination means 62 determines the position of the person to be recognized based on the acquired result. The direction determining means 63 determines the direction from the position of the person.

  The voice recognition function 70 includes a specific user dictionary setting unit 71, a voice recognition (texting) unit 72, an individual user dictionary DB (Data Base: database) 73, and a recognition result DB (Data Base: database) 74. The specific user dictionary setting means 71 performs a setting to use a DB dedicated to a person to be recognized in the individual user dictionary DB 73. The voice recognition (text conversion) means 72 performs voice recognition (text conversion) of the voice data obtained from the microphone 12 using the set dictionary DB.

  The sentence end determination function 80 includes a text acquisition unit 81 and a sentence end determination unit 82. The text acquisition unit 81 acquires the text information transmitted from the voice recognition function 70. The sentence end determination means 82 analyzes the obtained text information and determines whether or not the sentence is complete.

  Note that the face recognition function 30, mouth movement determination function 40, voice data input presence / absence determination function 50, directivity direction determination function 60, voice recognition function 70, and sentence end determination function 80 in the voice recognition system of this embodiment are a program. It is realizable with CPU which performs a process based on this. Further, a DB (Data Base) included in each function is stored in a storage device such as an HDD (Hard Disk Drive).

  Hereinafter, the operation of the voice recognition system of this embodiment will be described. FIG. 2 is a sequence diagram showing the operation of the embodiment of the speech recognition system according to the present invention. FIG. 3 is a flowchart showing the operation of the embodiment of the speech recognition system according to the present invention.

  When the power of the device 10 is turned on by the user (step S100), the camera 11 included in the device 10 captures the user and transmits an image to the face recognition function 30 and the mouth movement determination function 40 (step S100). S1). The face recognition function 30 performs face recognition by inputting an image obtained from the camera 11 (steps S101 and S2). Specifically, the DB collation unit 32 collates the feature amount of the image extracted by the feature amount extraction unit 31 with the face feature amount stored in advance in the face DB 33 to identify the user. If a person cannot be identified in face recognition (NO in step S102), face recognition is performed again (step S101).

  If the person can be identified in the face recognition (YES in step S102), the face recognition function 30 transmits the recognition result to the mouth movement determination function 40 and the voice recognition function 70 (step S3), and temporarily stops the face recognition. .

  An image obtained from the camera 11 is periodically transmitted to the mouth movement determination function 40 (step S5). The mouth state detection unit 41 detects the mouth state from the image obtained from the camera 11. The specific user DB setting 42 sets the mouth movement feature amount corresponding to the person recognized by the face recognition function 30 using the individual mouth movement database 43 (steps S103 and S4). The mouth movement determination means 44 at the time of utterance compares the feature quantity of the image with the feature quantity stored in the individual mouth movement database 43 to determine whether or not the user is speaking (step S104). Step S6). If it is determined that the user is not speaking, the face recognition function 30 is activated and face recognition is performed again (step S101).

  FIG. 4 is an explanatory diagram showing the operation of the mouth movement determination function 40 for a specific person. As shown in FIG. 4, the individual mouth movement database 43 includes an individual DB for each individual. In the example shown in FIG. 4, the individual mouth movement database 43 includes a database 43a dedicated to Mr. A and a database 43b dedicated to Mr. B. If the image obtained from the camera 11 is recognized as Mr. A in step S101, the mouth movement determination means 44 during utterance calculates the feature amount of the image and the feature amount stored in the database 43a dedicated to Mr. A. To determine if A is speaking.

  Mouth movement determination function 40 stores in advance mouth movement feature value data for each individual, and uses that data to determine whether the user is speaking. It is possible to determine with higher accuracy than to determine whether or not

  If it is determined in step S104 that the user is speaking, the mouth movement determination function 40 transmits the face recognition result obtained from the face recognition function 30 and the determination result of the mouth movement to the pointing direction determination function 60. (Step S7). The directivity direction determination function 60 determines the distance to the user and in what direction the directivity of the microphone 12 should be directed based on the transmitted result (step S105). Specifically, the search result acquisition unit 61 acquires the results of the face recognition function 30 and the mouth movement determination function 40. Based on the acquired result, the target person's image position determination means 62 determines the position and distance of the person to be recognized. Moreover, the direction determination means 63 determines a direction from a position. The determined pointing direction is notified to the device 10 (step S8). The device 10 automatically controls the microphone 12 to increase the directivity in that direction (step S106).

  If it is determined in step S104 that the user is speaking, the voice recognition function 70 sets a voice recognition target dictionary of a person who is a target of voice recognition (step S107). Specifically, since the user identified by the face recognition function 30 in this embodiment is Mr. A, the specific user dictionary setting unit 71 performs setting for using the dictionary dedicated to Mr. A in the individual user dictionary DB 73. .

  The device 10 transmits the voice data collected from the microphone 12 to the voice data input presence / absence determination function 50 (step S9). The audio data acquisition unit 51 acquires audio data from the device 10. The voice data presence / absence determining means 52 determines the presence / absence of voice data (step S108, step S10). For example, the sound data presence / absence determination unit 52 determines that sound data is included if the volume of the acquired sound data exceeds a predetermined value. If it is determined in step S108 that no audio data is contained, the face recognition function 30 is activated and the face recognition in step S101 is performed again.

  When it is determined in step S108 that voice data is included, the voice data input presence / absence determination function 50 transmits the voice data to the voice recognition function 70 that performs voice recognition (step S11). The voice recognition function 70 performs voice recognition on the transmitted voice data (steps S109 and S12).

  When the voice data input presence / absence determination function 50 determines that no voice data is contained, voice recognition is not performed, and therefore the possibility that the voice recognition function 70 performs unnecessary processing can be reduced.

  FIG. 5 is an explanatory diagram showing the operation of the voice recognition function 70 for a specific person. The individual user dictionary DB 73 stores voice-related features such as voice quality and words often used for each individual. In the example shown in FIG. 5, the individual user dictionary DB 73 includes a Mr. A dedicated dictionary DB 73a and a Mr. B dedicated dictionary DB 73b. In step S <b> 107, the specific user dictionary setting unit 71 is set to use a dictionary dedicated to Mr. A. Therefore, the voice recognition (text conversion) means 72 performs voice recognition (text conversion) of the voice data obtained from the microphone 12 using the Mr. A dedicated dictionary 73a (steps S109 and S12).

  Since the voice recognition function 70 performs voice recognition using voice-related features such as voice quality for each individual stored in advance and words often used, the voice recognition function 70 is more accurate than the case of voice recognition without specifying an individual. Voice recognition can be performed.

  The voice recognition function 70 accumulates voice-recognized (text-formatted) data (text information) in the recognition result DB 74 and transmits it to the sentence end determination function 80 (step S13). The text acquisition unit 81 acquires the text information transmitted from the voice recognition function 70. Then, the sentence end determination means 82 analyzes the obtained text information and determines whether or not the sentence is complete (step S14).

  FIG. 6 is an explanatory diagram showing the operation of the sentence end determination function 80. As shown in FIG. 6, the sentence end determination function 80 analyzes the text information obtained from the speech recognition function 70 and determines the end of the sentence. For example, when a word such as “Isao” or “Shimasho” is often used at the end of a sentence, it is determined that the word is the end of the sentence.

  In the determination of the sentence end determination function 80, if it is determined that the sentence is complete (YES in step S110), it is determined that the story of Mr. A is once ended, and the face recognition function is used to identify the next speaker. 30 is restarted and face recognition is performed again (step S101, step S15). In the determination by the sentence end determination function 80, if it is determined that the sentence is not complete (NO in step S1101), the process returns to step S106. Also, after recognizing the voice, even if there is no voice input for a certain period of time, it is determined that the person's utterance has ended, and the face recognition function 30 is restarted to identify the person who speaks next and the face recognition is performed again. (Step S101, Step S15).

  Since the sentence end determination function 80 can determine the end of the sentence, it can determine whether or not the speaker has finished speaking. And if the speaker decides that he has finished speaking, he recognizes the face again, and when another person begins to speak, he orients that person, so even if multiple people speak in a meeting, for example, the speaker is more accurate Can be directed to

  According to the present invention, it is determined whether or not speaking is performed using the mouth movement feature amount data stored for each individual in advance, and the directivity to a person who is highly likely to be speaking is improved. Since data can be collected, the accuracy of speech recognition can be improved. Further, since voice recognition is performed using a dedicated dictionary for each individual stored in advance, the accuracy of voice recognition can be improved. Further, when the end of the speaker's story is determined and it is determined that the speaker has ended, the face recognition is performed again and the directivity is controlled, so that the accuracy of speech recognition can be improved.

  FIG. 7 is a block diagram showing the main part of the speech recognition system according to the present invention. As shown in FIG. 7, the speech recognition system includes, as main components, a face recognition unit 1 that acquires an image of a user who is a target of speech recognition from a camera and identifies the user using the image; A mouth movement database that stores pre-stored individual mouth movement feature quantities, detects the mouth state of the user from the image, and corresponds to the user stored in the mouth movement database. The mouth movement determination means 2 for comparing whether the user is speaking by comparing the feature amount of the movement of the user and the feature amount of the mouth movement of the user obtained from the image, and When the determination is made, it is provided with a directivity direction determining means 3 for notifying the voice input means for acquiring the user's voice, and a voice recognition means 4 for acquiring the voice and performing voice recognition.

  In each of the above embodiments, a speech recognition system as shown in the following (1) to (4) is also disclosed.

(1) Face recognition means (for example, face) that acquires an image obtained by capturing a user (for example, user X) as a voice recognition target from a camera (for example, camera 11) and identifies the user using the image. Recognition function 30) and a mouth movement database (for example, individual mouth movement DB 43) that stores pre-stored feature values of mouth movements for each individual, and detects a user's mouth state from an image. Then, the feature value of the mouth movement corresponding to the user stored in the mouth movement database is compared with the feature value of the user's mouth movement obtained from the image to determine whether or not the user is speaking. It is used for mouth movement determination means (for example, mouth movement determination function 40) and voice input means (for example, microphone 12) for acquiring the user's voice when it is determined that the user is speaking. The user ’s location and directing the user Speech recognition system comprising Mesa causing directivity direction setting unit (e.g., pointing direction determining function 60) and speech recognition means for performing speech recognition to get the sound (e.g., voice recognition function 70) and.

(2) In the speech recognition system, the speech recognition means has an individual user dictionary database in which features relating to speech for each individual are stored in advance, and based on features relating to speech corresponding to users stored in the individual user dictionary database. The voice recognition of the user acquired from the voice input means may be performed.

(3) The speech recognition system acquires text information obtained by recognition by the speech recognition means, analyzes the text information, determines whether the sentence is complete, and determines that the sentence is complete, A sentence end determination unit (for example, a sentence end determination function 80) that causes the recognition unit to recognize a face may be provided.

(4) The voice recognition system includes voice input presence / absence determination means (for example, voice data input presence / absence determination function 50) for determining whether voice data is acquired from the voice input means, and when voice data is not acquired, The voice recognition unit may be configured not to perform voice recognition, and the face recognition unit may be configured to identify a user using an image.

  The present invention is applicable to speech recognition using a smartphone, speech recognition in a video conference, speech recognition in a meeting or lecture, and the like.

DESCRIPTION OF SYMBOLS 10 Apparatus 20 Network 30 Face recognition function 40 Mouth movement determination function 50 Speech data input presence / absence determination function 60 Directional direction determination function 70 Speech recognition function 80 Sentence end determination function

Claims (6)

Face recognition means for acquiring an image of a user who is a target of speech recognition from a camera and identifying the user using the image;
It has a mouth movement database that stores pre-stored mouth movement feature values for each individual, detects a user's mouth state from the image, and stores the mouth movement database in the user stored in the mouth movement database. Mouth movement determination means for comparing the corresponding mouth movement feature quantity with the user mouth movement feature quantity obtained from the image and judging whether the user is speaking;
When it is determined that the user is speaking, a direction-of-direction determining unit that notifies the user's position to a voice input unit for acquiring the user's voice;
A speech recognition system comprising speech recognition means for acquiring the speech and performing speech recognition. Voice recognition means
The user's voice acquired from the voice input unit based on the voice-related characteristics stored in the individual user dictionary database, having an individual user dictionary database storing the voice-related characteristics for each individual in advance The speech recognition system according to claim 1, wherein recognition is performed. Acquires text information obtained by the speech recognition means, analyzes the text information to determine whether the sentence is complete, and if it is determined that the sentence is complete, causes the face recognition means to recognize the face The speech recognition system according to claim 1, further comprising sentence end determination means. Comprising voice input presence / absence determining means for determining whether voice data is acquired from the voice input means;
The voice according to any one of claims 1 to 3, wherein when voice data is not acquired, the voice recognition means does not perform voice recognition, and the face recognition means specifies a user using an image. Recognition system. Obtain an image of the user who is the target of speech recognition from the camera, identify the user using the image,
It has a mouth movement database that stores pre-stored mouth movement feature values for each individual, detects a user's mouth state from the image, and stores the mouth movement database in the user stored in the mouth movement database. Compare the corresponding mouth movement feature quantity with the user mouth movement feature quantity obtained from the image, determine whether the user is speaking,
When it is determined that the user is speaking, the position of the user is notified to the voice input means for acquiring the voice of the user,
A voice recognition method characterized by acquiring the voice and performing voice recognition. On the computer,
A face recognition process for acquiring an image of a user who is a target of speech recognition from a camera and identifying the user using the image;
It has a mouth movement database that stores pre-stored mouth movement feature values for each individual, detects a user's mouth state from the image, and stores the mouth movement database in the user stored in the mouth movement database. Mouth movement determination processing for comparing the corresponding mouth movement feature amount with the user mouth movement feature amount obtained from the image, and determining whether the user is speaking;
When it is determined that the user is speaking, a direction determination process for notifying the user's position to voice input means for acquiring the user's voice;
A voice recognition program for executing voice recognition processing for acquiring voice and performing voice recognition.

Images