JP2022059957A - Voice recognition device - Google Patents

Abstract

To provide a voice recognition device that can recognize utterance not using a vocal cord.SOLUTION: The voice recognition device includes: an imaging unit that acquires an image including a lip region of a speaker; a lip movement locus detection unit that detects a lip movement locus of the speaker from the image; a non-audible sound detection unit that detects a non-audible sound propagating in the air from a voice of the speaker; a frequency pattern extraction unit that analyzes frequency characteristics of the non-audible sound, and extracts a frequency pattern; a lip movement locus data storage unit that previously stores a correspondence between the lip movement locus and utterance content; a non-audible sound pattern storage unit that previously stores a correspondence between the frequency pattern of the non-audible sound and utterance content; an utterance candidate extraction unit that refers to the lip movement locus data storage unit to extract candidates for utterance content from the lip movement locus; an utterance determination unit that, in a case in which a plurality of candidates for utterance content are extracted, refers to the non-audible sound pattern storage unit to determine specific utterance content among the plurality of candidates for utterance content; and an output unit that outputs information about the determined utterance content.SELECTED DRAWING: Figure 1

Description

The present invention relates to a voice recognition device.

Many elderly people speak without using the vocal cords. Also, when speaking in a low voice, the utterance may be performed without using the vocal cords. As a method of knowing the utterance content of a person who does not use the vocal cords, a method of recognizing words from the movement of the mouth has been reported (for example, Patent Document 1).

Patent Document 1 describes a mouth shape that needs to be formed in advance when a predetermined sound is emitted based on the mouth shape information indicating the mouth shape of a speaker in a lip reading device that recognizes words based on the movement of the mouth. A first mouth shape detecting means for detecting a first mouth shape, which is a mouth shape different from the mouth shape corresponding to a vowel of a sound, and a second mouth shape, which is a mouth shape formed when one sound is finished to be emitted. A lip reading device including a recognition means for recognizing a word spoken by a speaker based on a first mouth shape and a second mouth shape is described.

Further, a method of detecting an utterance content from an inaudible sound emitted when speaking without using a vocal cord has been reported (for example, Patent Document 2).

In Patent Document 2, a stethoscope-type microphone is attached to the human body surface, and a method of collecting the vibration sound of the flesh conduction of the non-audible muttering sound that is tuned along with the speech behavior that does not use the regular lip movement of the vocal cords. Is disclosed.

However, Japanese has a plurality of utterances with the same lip movement, and there is a problem that only a limited number of words can be deciphered by the method of deciphering the lip movement with a camera as in Patent Document 1.

Further, the method described in Patent Document 2 has a problem that a dedicated microphone must be attached in advance and the scenes that can be used are limited.

Japanese Unexamined Patent Publication No. 2008-310382 International Publication No. 2004/021738

An object of the present invention is to provide a voice recognition device capable of recognizing the content of an utterance even when the speaker speaks without using the vocal cords.

The voice recognition device according to the embodiment of the present disclosure includes an imaging unit that acquires an image including a lip region during a speaker's utterance operation, a lip movement locus detection unit that detects the lip movement locus of the speaker from the image, and a lip movement locus detection unit. An inaudible sound detector that detects inaudible sound propagating in the air from the voice spoken by the speaker, a frequency pattern extractor that analyzes the frequency characteristics of the inaudible sound and extracts frequency patterns, and lip movement. The lip smacking data storage unit that stores the correspondence between the locus and the utterance content in advance, the non-audible sound pattern storage unit that stores the correspondence between the frequency pattern of the inaudible sound and the utterance content in advance, and the lip movement locus data storage. If the utterance candidate extraction unit extracts candidates for utterance content from the locus of lip movement and the utterance candidate extraction unit extracts multiple candidates for utterance content, refer to the non-audible sound pattern storage unit. It is characterized by having an utterance determination unit that determines a specific utterance content from a plurality of utterance content candidates, and an output unit that outputs information about the utterance content determined by the utterance determination unit.

In the above voice recognition device, it is preferable that the non-audible sound detection unit starts the detection of the non-audible sound of the speaker by using the start of lip movement of the speaker detected by the lip movement locus detection unit as a trigger.

In the above speech recognition device, the non-audible sound detection unit preferably detects sound waves of 20 kHz or more and 70 kHz or less as non-audible sounds.

In the above voice recognition device, it is preferable that the utterance determination unit determines the utterance content based on the presence / absence of a peak in the frequency pattern and the position of the peak generated in a specific frequency band.

In the above voice recognition device, a plurality of utterance contents having substantially the same lip movement locus may include at least two of "na", "ta", and "da".

In the above voice recognition device, a plurality of utterance contents in which the locus of lip movement is substantially the same may include "" and "chi".

In the above voice recognition device, a plurality of utterance contents having substantially the same lip movement locus may include "a" and "ha".

According to the voice recognition device of the present invention, even when the speaker speaks without using the vocal cords, the content of the utterance can be recognized.

It is a block diagram of the voice recognition apparatus which concerns on embodiment of this disclosure. (A) is an example of the contour of the face recognized by the face image recognition unit, and (b) is an example of the contour of the mouth included in the contour of the face of (a). It is a figure which shows the locus of lip movement when uttering "na", "ta", and "da", (a) shows the locus of lip movement in the y direction, and (b) is the lip movement in the x direction. Represents the trajectory of. (A) to (c) are frequency spectra of the voice when uttering "na", "ta", and "da", respectively. It is a flowchart for demonstrating the operation procedure of the voice recognition apparatus which concerns on embodiment of this disclosure. It is a figure which shows the locus of lip movement when uttering "shi" and "chi", (a) shows the locus of lip movement in the y direction, and (b) shows the locus of lip movement in the x direction. (A) and (b) are frequency spectra of voice when uttering "shi" and "chi", respectively. It is a figure which shows the locus of lip movement when uttering "a" and "ha", (a) shows the locus of lip movement in the y direction, and (b) shows the locus of lip movement in the x direction. (A) and (b) are frequency spectra of voice when uttering "a" and "ha", respectively. FIG. 3 is a schematic configuration diagram of a conversation system using the voice recognition device according to the first embodiment. It is a block diagram of the voice recognition apparatus which concerns on Example 1. FIG. It is a block diagram of the voice recognition apparatus which concerns on the modification of Example 1. FIG. FIG. 3 is a schematic configuration diagram of an interpreter device using the voice recognition device according to the second embodiment. It is a block diagram of the voice recognition apparatus which concerns on Example 2. FIG. FIG. 3 is a schematic configuration diagram of a voice device operation system using the voice recognition device according to the third embodiment. It is a block diagram of the voice recognition apparatus which concerns on Example 3. FIG.

Hereinafter, the voice recognition device according to the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to those embodiments and extends to the inventions described in the claims and their equivalents.

FIG. 1 shows a block diagram of the voice recognition device 1001 according to the embodiment of the present disclosure. The voice recognition device 1001 includes an imaging unit 1, a lip movement locus detection unit 2, an inaudible sound detection unit 3, a frequency pattern extraction unit 4, a lip movement locus data storage unit 5, and an inaudible sound pattern storage unit 6. It has a speech candidate extraction unit 7, a speech determination unit 8, an output unit 9, and a face image recognition unit 10. An information terminal such as a smartphone or a tablet terminal can be used as the voice recognition device 1001. However, the present invention is not limited to such an example, and can be realized as an embedded module using a single board computer. Alternatively, the voice recognition device 1001 may be arranged on the server, and the data acquired by the image pickup unit 1 and the inaudible sound detection unit 3 may be transmitted to the server. The image pickup unit 1 is composed of a camera, and the inaudible sound detection unit 3 is composed of a microphone. The lip motion locus data storage unit 5 and the inaudible sound pattern storage unit 6 are composed of a hard disk or a semiconductor memory. The lip motion locus detection unit 2, the frequency pattern extraction unit 4, the utterance candidate extraction unit 7, the utterance determination unit 8, the output unit 9, and the face image recognition unit 10 are provided in the voice recognition device 1001 including a CPU, ROM, RAM, and the like. It is realized as software (program) by the computer.

The image pickup unit 1 is a camera and includes a CMOS (Complementary Metal Oxide Semiconductor) type or CCD (Charge Coupled Device) type image sensor. The image pickup unit 1 acquires an image including the lip region during the speech operation of the speaker, and supplies the captured image to the face image recognition unit 10 for each frame. As the camera, a camera provided in advance in an information terminal such as a smartphone or a tablet terminal can be used, and an external camera can also be used.

The face image recognition unit 10 has a function of identifying and automatically tracking the contours of the speaker's face and lips by a built-in application program for face recognition. As a result, even if the speaker moves within the imaging range of the imaging unit 1, the speaker's face image can be captured.

The lip movement locus detection unit 2 detects the lip movement locus of the speaker from the image acquired by the imaging unit 1. FIG. 2A is an example of the contour of the face recognized by the face image recognition unit 10, and FIG. 2B is an example of the contour of the mouth included in the contour of the face of FIG. 2A. As shown in FIG. 2A, the face image recognition unit 10 can determine the positions of the contours of the face 21, the eyebrows 22, the eyes 23, the nose 24, and the mouth 25. As shown in FIG. 2B, the lips open and close in the vertical direction (y direction) and expand and contract in the horizontal direction (x direction) due to the utterance. Therefore, the feature points for showing the movement of the lips are the lower end y1 of the upper lip, the upper end y2 of the lower lip, the left end portion x1 of the lips, and the right end portion x2 of the lips. Further, the feature amount of the movement of the lips was defined as a temporal change of the vertical distance (Δy = y1-y2) and a temporal change of the horizontal distance (Δx = x2-x1).

The lip movement locus data storage unit 5 stores in advance the correspondence between the lip movement locus (speech lip movement profile) and the utterance content. FIG. 3 is a diagram showing the locus of lip movement when uttering “na”, “ta”, and “da”, and FIG. 3 (a) shows the locus of lip movement in the y direction, and FIG. 3 (b). ) Represents the locus of lip movement in the x direction. The horizontal axis of FIGS. 3A and 3B is the time (seconds) from the start of lip movement. The vertical axis of FIG. 3A is the vertical distance Δy (mm), and the vertical axis of FIG. 3B is the horizontal distance Δx (mm). In FIG. 3A, Lny, Lty, and Ldy represent the locus of lip movement in the y direction when uttering “na”, “ta”, and “da”, respectively. Further, in FIG. 3B, Lnx, Ltx, and Ldx represent the locus of lip movement in the x direction when uttering “na”, “ta”, and “da”, respectively. In addition to the above example, the lip movement locus data storage unit 5 stores in advance the correspondence between the lip movement locus and the utterance content in various utterances. The lip movement locus data storage unit 5 may store a learning model generated by machine learning using artificial intelligence (AI) as to which speech content the feature amount of the lip movement locus is close to.

The utterance candidate extraction unit 7 extracts candidates for utterance content from the lip movement locus with reference to the lip movement locus data storage unit 5. In the examples shown in FIGS. 3 (a) and 3 (b), three curves (Lny, Lty,) representing the temporal change of lip movement in the y direction when uttering “na”, “ta”, and “da” are performed. Ldy) draws almost the same locus, and three curves (Lnx, Ltx, Ldx) representing the temporal change of lip movement in the x direction draw almost the same locus. Therefore, the curve representing the temporal change of lip movement in the y direction is similar to any of the three curves (Lny, Ly, Ldy), and the curve representing the temporal change of lip movement in the x direction is When an utterance showing a curve similar to any of the three curves (Lnx, Ltx, Ldx) is detected, the utterance content may be any of "na", "ta", and "da". As you can see, it is not possible to identify which of these utterances. Therefore, in such a case, three candidates for the utterance content are extracted.

The inaudible sound detection unit 3 detects the inaudible sound propagating in the air from the voice when the speaker speaks. As the inaudible sound detection unit 3, a MEMS (Micro Electro Mechanical Systems) microphone built in a smartphone or tablet terminal can be used. By using a MEMS microphone, it is possible to detect utterances including the frequency band of inaudible sounds. When the non-audible sound band is cut in order to reduce noise in the MEMS microphone built in the terminal such as a smartphone, such a band limitation may be released. Instead of using a microphone provided in advance in a smartphone or the like, a microphone capable of detecting inaudible sound may be externally attached. The inaudible sound detection unit 3 preferably detects sound waves of 20 kHz or more and 70 kHz or less as inaudible sounds.

It is preferable that the non-audible sound detection unit 3 starts the detection of the non-audible sound of the speaker by using the start of lip movement of the speaker detected by the lip movement locus detection unit 2 as a trigger. Inaudible sounds may be generated not only when the speaker speaks, but also when the speaker moves his / her body, which causes noise. Therefore, the timing at which the speaker starts speaking is not set. It may be difficult to detect only from audible sounds. Therefore, it is preferable that the non-audible sound detection unit 3 starts the detection of the non-audible sound after the lip movement locus detection unit 2 detects that the speaker's lips have started to move. By doing so, it is possible to accurately detect the inaudible sound generated by the speaker's utterance.

The frequency pattern extraction unit 4 analyzes the frequency characteristics of the inaudible sound and extracts the frequency pattern. FIG. 4 (a) is a frequency spectrum of the voice when "na" is spoken, FIG. 4 (b) is a frequency spectrum of the voice when "ta" is spoken, and FIG. 4 (c) is. , It is the frequency spectrum of the voice when uttering "da". In FIGS. 4A to 4C, the horizontal axis represents frequency (kHz) and the vertical axis represents power (dB). Even when the utterance is made without using the vocal cords, the frequency distribution (frequency pattern) in the inaudible region appears differently depending on how the tongue is used and how the throat is exhaled. By utilizing this difference in frequency distribution, it is possible to identify utterances that cannot be specified by lip smacking.

The non-audible sound pattern storage unit 6 stores in advance the correspondence between the frequency pattern of the non-audible sound and the utterance content. That is, the non-audible sound pattern storage unit 6 determines whether or not a peak occurs at a specific frequency as a feature point in the frequency pattern for identifying each of a plurality of utterances having substantially the same lip movement trajectory, and a peak. When this occurs, it stores information about the position of the peak that occurs in a specific frequency band. For example, the non-audible sound pattern storage unit 6 stores the presence / absence of a peak in each of the non-audible sound frequency patterns of “na”, “ta”, and “da” and the position of the peak generated in a specific frequency band. It is preferable to have. Specifically, as shown in FIG. 4A, when the utterance "na" is spoken, the entire tongue is lightly pressed against the upper jaw, so that no clear peak appears in the frequency pattern in the range of 20 kHz to 30 kHz. Further, as shown in FIG. 4 (b), when the utterance "ta" is spoken, the peak Pt appears in the frequency pattern in the range of 25 kHz to 30 kHz because the tip of the tongue is strongly applied to the upper jaw. Furthermore, as shown in FIG. 4 (c), when uttering "da", the tip of the tongue touches the upper jaw lighter than in the case of "ta" due to voiced sound, so the frequency pattern is 20 kHz to 25 kHz, which is lower than "ta". The peak Pd appears in the range of. As described above, in the non-audible sound pattern storage unit 6, when the utterance content is "na", no clear peak appears in the frequency pattern in the range of 20 kHz to 30 kHz, and when the utterance content is "ta", the frequency. It is memorized that the peak appears in the range of 25 kHz to 30 kHz in the pattern, and the peak appears in the range of 20 kHz to 25 kHz in the frequency pattern when the utterance content is "da". As described above, the non-audible sound pattern storage unit 6 determines whether or not a peak occurs at a specific frequency in the non-audible sound frequency pattern for each of a plurality of utterances having substantially the same lip movement trajectory, and the peak is When it occurs, information about which frequency band the peak occurs is stored in advance. However, in addition to these, the non-audible sound pattern storage unit 6 has substantially the same lip movement trajectory and a non-audible sound frequency pattern, such as "shi" and "chi", and "a" and "ha". Also remembers other combinations of different utterances.

When the utterance candidate extraction unit 7 extracts a plurality of utterance content candidates, the utterance determination unit 8 refers to the non-audible sound pattern storage unit 6 and selects a specific utterance content from the plurality of utterance content candidates. decide. For example, when the utterance candidate extraction unit 7 extracts three utterance content candidates "na", "ta", and "da", the above three utterance contents are referred to with reference to the non-audible sound pattern storage unit 6. Determine a specific utterance content from the candidates. As described above, when the lip movement locus detected by the lip movement locus detection unit 2 has a curve similar to that in FIGS. 3A and 3B, the utterance candidate extraction unit 7 stores the lip movement locus data. With reference to Part 5, "na", "ta", and "da" are extracted as candidates for utterance content from the locus of lip movement. Next, the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6 and uses the detected frequency pattern of the inaudible sound as three candidates for the utterance content (“na”, “ta”, “da”). By collating with each frequency pattern, a specific utterance content is determined from the three utterance content candidates.

The utterance determination unit 8 can determine the utterance content based on the presence / absence of a peak that occurs in a specific frequency band in the frequency pattern and the position of the peak. For example, as shown in FIG. 4A, when a power peak is not detected in a frequency in the range of 20 kHz to 30 kHz, it can be determined that the detected utterance is “na”. Further, as shown in FIG. 4B, when the peak power Pt is detected at a frequency in the range of 25 kHz to 30 kHz, it can be determined that the detected utterance is “”. Alternatively, as shown in FIG. 4C, when the peak power Pd is detected at a frequency in the range of 20 kHz to 25 kHz, it can be determined that the detected utterance is “da”. As described above, in the utterance determination unit 8, when the utterance candidate extraction unit 7 extracts three utterance content candidates (“na”, “ta”, “da”), the non-audible sound pattern storage unit 6 With reference to, one of "na", "ta", and "da" is determined as a specific utterance content from the three utterance content candidates.

When the utterance candidate extraction unit 7 extracts one utterance content candidate, the utterance determination unit 8 may determine the candidate as the utterance content uttered by the speaker without referring to the inaudible frequency pattern. can. In this case, since the step of referring to the inaudible frequency pattern can be omitted, the content of the utterance uttered by the speaker can be quickly determined, and the processing load in the voice recognition device 1001 can be reduced. can. However, if there is no problem in terms of ability, the utterance determination unit 8 may compare the detected voice frequency pattern with the frequency pattern stored in the inaudible sound pattern storage unit 6. This makes it possible to increase the reliability of this pronunciation determination.

The output unit 9 outputs information regarding the utterance content determined by the utterance determination unit 8. When a display device is connected to the output unit 9, the detected utterance content can be displayed as character information on the screen of the display device. Further, when the voice reproduction device is connected to the output unit 9, the detected utterance content can be output as voice. For example, in addition to the screen display, audio output from an earphone or the like may be used together.

Next, the operation procedure of the voice recognition device according to the present embodiment will be described. FIG. 5 is a flowchart for explaining the operation procedure of the voice recognition device according to the embodiment of the present disclosure. First, in step S101, the camera, which is the image pickup unit 1, is operated. The camera acquires an image including the lip region during the speaker's speech movement.

Next, in step S102, the face image recognition unit 10 identifies the contours of the speaker's face and lips.

Next, in step S103, the lip movement locus detection unit 2 detects the locus of the speaker's lip movement from the image captured by the camera.

Next, in step S104, the utterance candidate extraction unit 7 refers to the lip movement locus data storage unit 5 that previously stores the correspondence between the lip movement locus and the utterance content, and is a candidate for the utterance content from the lip movement locus. To extract.

On the other hand, after the camera is activated and it is detected that the speaker's lips have started to move, the inaudible sound sensor, which is the inaudible sound detection unit 3, is activated in step S105, and the speaker speaks. Detects inaudible sounds propagating in the air from the voice.

Next, in step S106, the frequency pattern extraction unit 4 analyzes the frequency characteristics of the inaudible sound, and extracts the frequency pattern in step S107.

Next, in step S108, the utterance candidate extraction unit 7 determines whether the number of utterance candidates extracted in step S104 is a plurality or one. When there is only one utterance candidate, in step S109, the utterance determination unit 8 determines the utterance by the speaker as a single utterance candidate. An example of a case where there is only one utterance candidate is, for example, a vowel. In this case, the content of the utterance can be determined only by the locus of lip movement. Therefore, when there is only one utterance candidate, it is not necessary to refer to the frequency pattern of the inaudible sound, so that the utterance content can be determined efficiently. However, if there is no problem in terms of ability, the utterance determination unit 8 may compare the detected voice frequency pattern with the frequency pattern stored in the inaudible sound pattern storage unit 6. This makes it possible to increase the reliability of this pronunciation determination.

On the other hand, when the utterance candidate extraction unit 7 extracts a plurality of utterance content candidates, in step S110, the utterance determination unit 8 stores in advance the correspondence between the frequency pattern of the inaudible sound and the utterance content. With reference to the pattern storage unit 6, a specific utterance content is determined based on a frequency pattern from a plurality of utterance content candidates.

Next, in step S111, the output unit 9 outputs the determined utterance content.

In the above description, the combination of "na", "ta", and "da" is exemplified as a plurality of utterance candidates extracted from the lip movement locus, but the present invention is not limited to such an example. That is, when a plurality of utterance contents having substantially the same lip movement trajectory are two combinations of "na", "ta", and "da", one utterance is made from the two combinations. You may decide the content. Furthermore, as another example of multiple utterance candidates, there are a combination of "shi" and "chi", and a combination of "a" and "ha", and the method of determining a specific utterance from these combinations is described below. explain.

First, a case where a plurality of utterance candidates are a combination of "shi" and "chi" will be described. FIG. 6 is a diagram showing the locus of lip movement when uttering “shi” and “chi”, FIG. 6 (a) shows the locus of lip movement in the y direction, and FIG. 6 (b) shows the locus of lip movement in the x direction. Represents the trajectory of lip movement. In FIG. 6A, Lsy and Lcy follow the locus of lip movement (Δy = y1-y2) in the y direction shown in FIG. 2B when uttering “shi” and “chi”, respectively. show. Further, in FIG. 6B, Lsx and Lcx represent the locus of lip movement (Δx = x2-x1) in the x direction when uttering “shi” and “chi”, respectively.

The utterance candidate extraction unit 7 extracts candidates for utterance content from the lip movement locus with reference to the lip movement locus data storage unit 5. In the examples shown in FIGS. 6 (a) and 6 (b), the curves Lsy and Lcy representing the temporal change of lip movement in the y direction when uttering “shi” and “chi” are substantially the same curve, and , Lsx and Lcx, which represent the temporal change of lip movement in the x direction, are almost the same curve. The lip smacking locus detection unit 2 detects the lip smacking locus in the y direction similar to the curve shown in FIG. 6 (a), and the lip smacking locus in the x direction is the curve shown in FIG. 6 (b). If they are similar, the lip movement candidate extraction unit 7 refers to the lip movement locus data storage unit 5 and extracts “shi” and “chi” as candidates for the speech content from the lip movement locus. Therefore, in this case, it is known that the utterance content is either "shi" or "chi", but it is not possible to specify which of these utterances. Therefore, in such a case, two candidates for the utterance content are extracted.

The frequency pattern extraction unit 4 analyzes the frequency characteristics of the inaudible sound and extracts the frequency pattern. FIG. 7 (a) is a frequency spectrum of the voice when "shi" is spoken, and FIG. 7 (b) is a frequency spectrum of the voice when "chi" is spoken.

The non-audible sound pattern storage unit 6 stores in advance the correspondence between the frequency pattern of the non-audible sound and the utterance content. For example, as shown in FIG. 7A, when the utterance “” is spoken, no clear peak appears in the frequency pattern near 40 kHz. On the other hand, as shown in FIG. 7B, when "chi" is spoken, the peak Pc appears in the frequency pattern near 40 kHz by pressing the center of the tongue against the upper jaw. In this way, the non-audible sound pattern storage unit 6 stores in advance the correspondence between the frequency patterns of the non-audible sounds of "" and "chi" and the utterance content.

Next, the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6, and uses the detected frequency pattern of the inaudible sound as the frequency pattern of each of the two utterance content candidates (“” and “chi”). By collating with, a specific utterance content is determined from the two utterance content candidates.

The utterance determination unit 8 can determine the utterance content based on the presence / absence of a peak in the frequency pattern and the position of the peak generated in a specific frequency band. As shown in FIG. 7A, when the power peak is not detected at a frequency near 40 kHz, it can be determined that the detected utterance is “”. Further, as shown in FIG. 7B, when the power peak Pc is detected in the vicinity of 40 kHz, it can be determined that the detected utterance is “chi”. As described above, when the utterance candidate extraction unit 7 extracts two utterance content candidates (“shi” and “chi”), the utterance determination unit 8 refers to the non-audible sound pattern storage unit 6. One of "shi" and "chi" is determined as a specific utterance content from the two utterance content candidates.

Next, a case where a plurality of utterance candidates are a combination of "a" and "ha" will be described. FIG. 8 is a diagram showing the locus of lip movement when uttering “a” and “ha”, FIG. 8 (a) shows the locus of lip movement in the y direction, and FIG. 8 (b) shows the locus of lip movement in the x direction. Represents the trajectory of lip movement. In FIG. 8 (a), Lay and Lhy follow the locus of lip movement (Δy = y1-y2) in the y direction shown in FIG. 2 (b) when uttering “a” and “ha”, respectively. show. Further, in FIG. 8 (b), Lax and Lhx represent the locus of lip movement (Δx = x2-x1) in the x direction when uttering “a” and “ha”, respectively.

The utterance candidate extraction unit 7 extracts candidates for utterance content from the lip movement locus with reference to the lip movement locus data storage unit 5. In the examples shown in FIGS. 8A and 8B, the curves Lay and Ly, which represent the temporal change of lip movement in the y direction when "a" and "ha" are spoken, are substantially the same curve, and , The curves Lax and Lhx representing the temporal change of lip movement in the x direction are almost the same curve. The lip smacking locus detection unit 2 detects the lip smacking locus in the y direction similar to the curve shown in FIG. 8 (a), and the lip smacking locus in the x direction is the curve shown in FIG. 8 (b). If they are similar, the speech candidate extraction unit 7 refers to the lip movement locus data storage unit 5 and extracts “a” and “ha” as candidates for the speech content from the lip movement locus. Therefore, in this case, it is known that the utterance content is either "a" or "ha", but it is not possible to specify which of these utterances. Therefore, in such a case, two candidates for the utterance content are extracted.

The frequency pattern extraction unit 4 analyzes the frequency characteristics of the inaudible sound and extracts the frequency pattern. FIG. 9 (a) is a frequency spectrum of the voice when "a" is spoken, and FIG. 9 (b) is a frequency spectrum of the voice when "ha" is spoken.

The non-audible sound pattern storage unit 6 stores in advance the correspondence between the frequency pattern of the non-audible sound and the utterance content. For example, as shown in FIG. 9A, when the utterance "A" is spoken, the entire tongue is lightly pressed against the upper jaw, so that no clear peak appears in the frequency pattern near 20 kHz. Further, as shown in FIG. 9B, when "ha" is spoken, the peak Ph appears in the frequency pattern near 20 kHz by pressing the center of the tongue against the upper jaw. In this way, the non-audible sound pattern storage unit 6 stores in advance the correspondence between the frequency patterns of the non-audible sounds of "a" and "ha" and the utterance content.

Next, the utterance determination unit 8 refers to the inaudible sound pattern storage unit 6, and uses the detected frequency pattern of the inaudible sound as the frequency pattern of each of the two utterance content candidates (“a” and “ha”). By collating with, a specific utterance content is determined from the two utterance content candidates.

The utterance determination unit 8 can determine the utterance content based on the presence / absence of a peak in the frequency pattern and the position of the peak generated in a specific frequency band. As shown in FIG. 9A, when the power peak is not detected in the frequency pattern near 20 kHz, it can be determined that the detected utterance is “A”. Further, as shown in FIG. 9B, when a power peak is detected in a frequency pattern near 20 kHz, it can be determined that the detected utterance is “ha”. As described above, when the utterance candidate extraction unit 7 extracts two utterance content candidates (“a” and “ha”), the utterance determination unit 8 refers to the non-audible sound pattern storage unit 6. One of "a" and "ha" is determined as a specific utterance content from the two utterance content candidates.

As described above, examples of multiple utterance contents in which the locus of lip movement is substantially the same are a combination of "na", "ta", and "da", a combination of "shi" and "chi", and "a". And "ha" are shown, but the present invention is not limited to such an example, and the voice according to the embodiment of the present disclosure is also applied to a combination of a plurality of other utterance contents having substantially the same lip movement trajectory. A recognition device can be used.

As described above, according to the voice recognition device according to the embodiment of the present disclosure, it is possible to determine utterances (murmurs) that do not use vocal cords in a non-contact manner, and rough candidates for predicted utterances are extracted from the lip movement locus. , It is possible to confirm the utterance by the speaker from the prediction candidates from the frequency pattern by the inaudible sound. Further, by combining the pattern determination based on lip movement and the determination based on inaudible sound, the accuracy of predicting the utterance content can be improved. According to the voice recognition device according to the embodiment of the present disclosure, it is possible to decipher the conversation of an elderly person spoken without using the vocal cords. In addition, it is possible to grasp the contents of a call made in a low voice without using a vocal cord in a vehicle or the like where silence is required. In this case, since the conversation can be performed by an inaudible sound, the conversation can be performed while protecting privacy or preventing information leakage. Further, since it is not necessary to equip the speaker with special equipment in advance, it can be used for various purposes.

[Example 1]
Next, the voice recognition device according to the first embodiment will be described. FIG. 10 is a schematic configuration diagram of a conversation system using the voice recognition device according to the first embodiment. When the speaker (120, 220) is an elderly person or the like, speaking without using the vocal cords may cause a problem that the contents spoken by each other cannot be heard, and as a result, communication cannot be performed well. The voice recognition device (100, 200) according to the first embodiment communicates by displaying the content of the words spoken by the self on the voice recognition device possessed by the self and showing it to the other party.

Here, it is assumed that the first speaker 120 has the first voice recognition device 100, and the second speaker 220 has the second voice recognition device 200. An information terminal such as a tablet terminal can be used for the first voice recognition device 100 and the second voice recognition device 200. The first speaker 120 can lower the first voice recognition device 100 from the neck to point the display unit 113 toward the second speaker 220, and the image pickup unit 101 can capture an image of the lip region 220 m of the second speaker 220. To do so. Similarly, in the second speaker 220, the second voice recognition device 200 is lowered from the neck and the display unit 213 is directed toward the first speaker 120, and the image pickup unit 201 is an image of the lip region 120 m of the first speaker 120. To be able to image.

First, a procedure will be described in which an image and an inaudible sound when the second speaker 220 speaks are analyzed by the first voice recognition device 100, and then the analysis result is displayed on the display unit 213 of the second voice recognition device 200. do. FIG. 11 is a block diagram of the voice recognition device (100, 200) according to the first embodiment. In addition to the configuration of the voice recognition device 1001 shown in FIG. 1, the first voice recognition device 100 and the second voice recognition device 200 include a receiving unit (111, 211), a transmitting unit (112, 212), and a display. The unit (113, 213) is provided. Other configurations are the same as the configuration of the voice recognition device 1001 shown in FIG.

The image pickup unit 101 of the first voice recognition device 100 acquires an image including the lip region 220 m when the second speaker 220 is speaking "What is today's gohan?" Without using the vocal cords. Further, the non-audible sound detection unit 103 detects the non-audible sound propagating in the air from the voice when the second speaker 220 speaks.

The first voice recognition device 100 determines from the acquired image of the lip region 220 m and the inaudible sound that the utterance content of the second speaker 220 is "What is today's gohan?", And outputs the determined result to the output unit 109. Outputs to the transmitter 112. The transmission unit 112 transmits information regarding the utterance content of the second speaker 220 to the reception unit 211 of the second voice recognition device 200.

The receiving unit 211 of the second voice recognition device 200 receives the information regarding the utterance content and transmits it to the display unit 213. Based on the acquired information, the display unit 213 displays "What is today's gohan?" On the display screen. By visually recognizing the display unit 213 of the second voice recognition device 200, the first speaker 120 recognizes that the content spoken by the second speaker 220 is "What is today's gohan?". At this time, by displaying on the display unit 213 that the second speaker 220 has finished speaking and is waiting for a reply from the first speaker 120, it is easy to take the timing for the first speaker 120 to speak. You may try to do it.

Next, the procedure for displaying the analysis result on the display unit 113 of the first voice recognition device 100 after the image and the inaudible sound when the first speaker 120 speaks are analyzed by the second voice recognition device 200. explain. The image pickup unit 201 of the second voice recognition device 200 acquires an image including the lip region 120 m when the first speaker 120 speaks "Curry" without using the vocal cords. Further, the non-audible sound detection unit 203 detects the non-audible sound propagating in the air from the voice when the first speaker 120 speaks.

The second voice recognition device 200 determines that the utterance content of the first speaker 120 is "curry" from the acquired image of the lip region 120 m and the inaudible sound, and the output unit 209 transmits the determined result. Output to 212. The transmission unit 212 transmits information regarding the utterance content of the first speaker 120 to the reception unit 111 of the first voice recognition device 100.

The receiving unit 111 of the first voice recognition device 100 receives the information regarding the utterance content and transmits it to the display unit 113. The display unit 113 displays "Curry" on the display screen based on the acquired information. The second speaker 220 recognizes that the content spoken by the first speaker 120 is "curry" by visually recognizing the display unit 113 of the first voice recognition device 100.

As described above, by using the voice recognition device according to the first embodiment, even when the first speaker 120 and the second speaker 220 speak without using the vocal cords, it is possible to grasp each other's utterance contents. You can and can communicate.

Next, as a modification of the first embodiment, after the first voice recognition device 100'acquires information on the image and the inaudible sound when the second speaker 220 speaks, the acquired information is used as the second voice. A procedure of transmitting to the recognition device 200', analyzing with the second voice recognition device 200', and displaying the analysis result will be described. FIG. 12 shows a block diagram of the voice recognition device according to the modified example of the first embodiment.

The image pickup unit 101 of the first voice recognition device 100'acquires an image including the lip region 220 m when the second speaker 220 is speaking "What is today's gohan?" Without using the vocal cords. Further, the non-audible sound detection unit 103 detects the non-audible sound propagating in the air from the voice when the second speaker 220 speaks.

The image data acquired by the image pickup unit 101 of the first voice recognition device 100'and the non-audible sound data acquired by the inaudible sound detection unit 103 are supplied to the transmission unit 112, and the transmission unit 112 transfers these data to the second unit. It is transmitted to the receiving unit 211 of the voice recognition device 200'.

The second voice recognition device 200'determines that the utterance content of the second speaker 220 is "What is today's gohan?" From the image and the inaudible sound of the lip region 220m received by the receiving unit 211. The result is displayed on the display unit 213.

By visually recognizing the display unit 213 of the second voice recognition device 200', the first speaker 120 recognizes that the content spoken by the second speaker 220 is "What is today's gohan?". At this time, by displaying on the display unit 213 that the second speaker 220 has finished speaking and is waiting for a reply from the first speaker 120, it is easy to take the timing for the first speaker 120 to speak. You may try to do it.

Next, after the content spoken by the first speaker 120 is acquired by the second voice recognition device 200', the acquired information is transmitted to the first voice recognition device 100', and the first voice recognition device 100' The procedure for analyzing and displaying is explained in.

The image pickup unit 201 of the second voice recognition device 200'acquires an image including the lip region 120 m when the first speaker 120 speaks "Curry" without using the vocal cords. Further, the non-audible sound detection unit 203 detects the non-audible sound propagating in the air from the voice when the first speaker 120 speaks.

The image data acquired by the image pickup unit 201 of the second voice recognition device 200'and the non-audible sound data acquired by the inaudible sound detection unit 203 are supplied to the transmission unit 212, and the transmission unit 212 transfers these data to the first transmission unit 212. It is transmitted to the receiving unit 111 of the voice recognition device 100'.

The first voice recognition device 100'determines that the utterance content of the first speaker 120 is "curry" from the image of the lip region 120 m and the inaudible sound received by the receiving unit 111, and displays the determined result. It is displayed on the unit 113.

By visually recognizing the display unit 113 of the first voice recognition device 100', the second speaker 220 recognizes that the content spoken by the first speaker 120 is "curry".

As described above, by using the voice recognition device according to the modified example of the first embodiment, even when the first speaker 120 and the second speaker 220 speak without using the vocal cords, the other party's utterance contents can be grasped by each other. Can and can communicate.

[Example 2]
Next, the voice recognition device according to the second embodiment will be described. FIG. 13 is a schematic configuration diagram of an interpreter using the voice recognition device 1002 according to the second embodiment. The caregiver 30 tries to grasp the emotion of the elderly person 40 who is a speaker from the conversation content, but when the elderly person 40 speaks without using the vocal cords, he / she cannot hear the uttered content and grasps the emotion. Can be difficult. The voice recognition device 1002 according to the second embodiment recognizes the utterance content of the elderly person 40 by displaying the utterance content of the elderly person 40 on the display unit 13 without using the vocal cords.

FIG. 14 is a block diagram of the voice recognition device 1002 according to the second embodiment. The voice recognition device 1002 according to the second embodiment includes a display unit 13 in addition to the voice recognition device 1001 shown in FIG. Other configurations are the same as those in the voice recognition device 1001.

The image pickup unit 1 of the voice recognition device 1002 acquires an image including the lip region 40 m when the elderly person 40 who is a speaker speaks without using the vocal cords. Further, the inaudible sound detection unit 3 detects the inaudible sound propagating in the air from the voice when the elderly person 40 speaks.

The voice recognition device 1002 determines the utterance content of the elderly 40 based on the acquired image including the lip region 40 m and the detected inaudible sound, and outputs the data related to the utterance content determined by the output unit 9 to the display unit 13. do. The display unit 13 displays the utterance content based on the acquired data.

According to the voice recognition device 1002 according to the second embodiment, even when the elderly person 40 speaks without using the vocal cords, the utterance content can be displayed on the display unit 13, so that the utterance content of the elderly person 40 can be displayed. Can be recognized.

[Example 3]
Next, the voice recognition device according to the third embodiment will be described. FIG. 15 is a schematic configuration diagram of a voice device operation system using the voice recognition device 1003 according to the third embodiment. In recent years, devices for operating home appliances by voice and controlling automobile navigation systems have been developed. These devices can be controlled by recognizing the received voice, but even if elderly people try to speak and operate without using vocal cords, these devices can emit without using such vocal cords. There is a problem that the received voice cannot be recognized. The voice recognition device 1003 according to the third embodiment recognizes the content spoken by the elderly person 50 without using the vocal cords, converts it into audible voice, and causes the device 60 such as a home appliance to recognize the voice converted into audible voice. Is.

FIG. 16 is a block diagram of the voice recognition device 1003 according to the third embodiment. The voice recognition device 1003 according to the third embodiment includes a voice reproduction unit 14 in addition to the voice recognition device 1001 shown in FIG. Other configurations are the same as those in the voice recognition device 1001.

The image pickup unit 1 of the voice recognition device 1003 acquires an image including the lip region 50 m when the elderly person 50 who is a speaker speaks without using the vocal cords. Further, the inaudible sound detection unit 3 detects the inaudible sound propagating in the air from the voice when the elderly person 50 speaks.

The voice recognition device 1003 determines the utterance content of the elderly person 50 based on the acquired image including the lip region 50 m and the detected inaudible sound, and the output unit 9 outputs the utterance content to the voice reproduction unit 14. The voice reproduction unit 14 reproduces the utterance content of the elderly person 50 as an audible sound, and the device 60 recognizes the reproduced audible sound and executes a predetermined control.

According to the voice recognition device 1003 according to the third embodiment, even when the elderly person 50 speaks without using the vocal cords, the utterance content can be converted into an audible sound and output from the voice reproduction unit 14. , The voice recognition accuracy in the device 60 can be improved.

1 Imaging unit 2 Lip movement locus detection unit 3 Inaudible sound detection unit 4 Frequency pattern extraction unit 5 Lip movement locus data storage unit 6 Non-audible sound pattern storage unit 7 Speech candidate extraction unit 8 Speech determination unit 9 Output unit 10 Face image recognition Section 13 Display section 14 Audio playback section

Claims (7)

An image pickup unit that acquires an image including the lip region during the speaker's utterance operation,
A lip movement locus detection unit that detects the locus of the speaker's lip movement from the image,
An inaudible sound detection unit that detects an inaudible sound propagating in the air from the voice when the speaker speaks,
A frequency pattern extraction unit that analyzes the frequency characteristics of the inaudible sound and extracts a frequency pattern,
A lip movement locus data storage unit that previously stores the correspondence between the lip movement locus and the utterance content,
An inaudible sound pattern storage unit that stores in advance the correspondence between the frequency pattern of the inaudible sound and the utterance content,
With reference to the lip movement locus data storage unit, an utterance candidate extraction unit that extracts utterance content candidates from the lip movement locus, and a speech candidate extraction unit.
When the utterance candidate extraction unit extracts a plurality of utterance content candidates, the utterance determination unit determines a specific utterance content from the plurality of utterance content candidates with reference to the inaudible sound pattern storage unit. When,
An output unit that outputs information about the utterance content determined by the utterance determination unit, and an output unit.
A speech recognition device characterized by having. The voice recognition device according to claim 1, wherein the non-audible sound detection unit starts detection of the non-audible sound of the speaker triggered by the start of lip movement of the speaker detected by the lip movement locus detection unit. .. The voice recognition device according to claim 1 or 2, wherein the non-audible sound detection unit detects sound waves of 20 kHz or more and 70 kHz or less as the non-audible sound. The voice recognition device according to any one of claims 1 to 3, wherein the utterance determination unit determines the utterance content based on the presence or absence of a peak in the frequency pattern and the position of the peak generated in a specific frequency band. .. The present invention according to any one of claims 1 to 4, wherein the plurality of speech contents having substantially the same lip movement trajectory include at least two of "na", "ta", and "da". Speech recognition device. The voice recognition device according to any one of claims 1 to 4, wherein the plurality of utterance contents having substantially the same lip movement trajectory include "shi" and "chi". The voice recognition device according to any one of claims 1 to 4, wherein the plurality of utterance contents having substantially the same lip movement trajectory include "a" and "ha".

Images