JP2018013549A - Speech content recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a speech content recognition device that creates intermediate language data on the basis of a shot image of a camera.SOLUTION: A voice recognition device 1 is configured to: when detecting speeches of users during operating in a standby mode, create connection lip motion data indicative of a motion pattern of a lip upon successively pronouncing two syllables, and tone pattern data indicative of a tone upon successively pronouncing the two syllables on the basis of voice data spoken by the user and image data in this instance; store the created connection lip motion data and the tone pattern data in a tone DB1c; when executing lip reading processing with a depression of a talk SW3 of a user as a trigger, determine a connection lip motion pattern from the image data at a time of the speech of the user; and decide the tone for each syllable from the determined connection lip motion pattern and data stored in the tone DB1c.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for recognizing the content of a user speaking from the movement pattern of the user's lips captured by a camera.

  Conventionally, as disclosed in Patent Document 1, by performing image recognition processing on the face image of the speaker, the utterance content of the utterer is specified, and text data and voice data corresponding to the utterance content are generated. There is technology (so-called lip reading technology). Such a lip reading technique is used for various applications such as character input.

JP2015-220684A

  By the way, the user's uttered voice includes not only text information indicating what is being spoken but also tone information on what tone is being spoken. However, in the conventional lip reading technology, a method for generating text data (hereinafter referred to as intermediate language data) including tone information included in a user's uttered voice has not been studied. Note that the tone information here is information indicating various parameters such as intonation (so-called intonation), the pitch of the user's voice, speaking speed, volume, and the like.

  The present invention has been made based on this situation, and an object thereof is to provide an utterance content recognition device that generates intermediate language data based on a captured image of a camera.

  In order to achieve the object, the present invention is an utterance content recognition device that executes processing for specifying the utterance content of a user with a predetermined user operation as a trigger, an operation reception unit (F2) that receives the user operation, and a microphone A voice acquisition unit (F3) that acquires the user's utterance voice as utterance voice data, and an image acquisition unit that sequentially acquires user images that are images taken by a camera arranged to shoot the user's face (F6) and a lip movement pattern that is a change pattern of the user's lip shape from the user image acquired by the image acquisition unit, and further, a character string corresponding to the user's utterance content based on the detected lip movement pattern A lip reading processing unit (F7) that generates an utterance character string and a tone pattern for uttering two syllables continuously based on the utterance voice data acquired by the voice acquisition unit A tone pattern specifying unit (F81) that specifies a lip, and a connected lip movement pattern that is a lip change pattern when the user utters two syllables continuously from a user image acquired by the image acquisition unit The linked lip motion pattern specified by the connected lip motion pattern specifying unit and the tone pattern specified by the tone pattern specifying unit are associated with the lip motion pattern specifying unit (F82) and the two syllables continuously uttered by the user. Using the learning processing unit (F8) that executes pattern learning processing that is processing stored in the tone database, the data stored in the tone database, and the user image used for generating the utterance character string, An intermediate language data generation unit (F9) for generating intermediate language data in which tone information is added to each syllable character constituting the sequence, and the learning processing unit When the user operation is not received, the learning process is sequentially executed in cooperation with the tone pattern specifying unit and the connected lip movement pattern specifying unit, and the intermediate language data generating unit is configured so that the operation receiving unit receives the user operation. The lip reading processing unit generates intermediate language data when the lip reading processing unit generates an utterance character string, and the intermediate language data generation unit generates a syllable character of the target character that constitutes the utterance character string. When determining the tone, the lip shape of the user when the syllable character and the target character located immediately before the target character are uttered from the plurality of connected lip movement patterns stored in the tone database. A connected lip movement pattern having a high degree of similarity to the change pattern is specified, and the tone of the target character is determined using the tone pattern associated with the specified connected lip movement pattern. And features.

  In the above configuration, when the operation receiving unit has not received a user operation corresponding to the execution instruction of the process for specifying the utterance content (in other words, in the input standby state), the learning processing unit However, the pattern learning process is sequentially executed in cooperation with the tone pattern specifying unit and the connected lip movement pattern specifying unit. That is, based on the user's daily conversation, learning is performed with the tone pattern of the user's utterance and the connected lip movement pattern for every two syllables. Therefore, the data stored in the tone database is data based on the user's actual speech.

  When the lip reading processing unit generates a character string corresponding to the movement of the user's lips (that is, an uttered character string) based on the reception of the above-described user operation by the operation receiving unit, the intermediate language data generating unit Then, intermediate language data in which the tone information is added to the utterance character string is generated from the data stored in the tone database and the user image used to generate the utterance character string.

  According to such an aspect, intermediate language data can be generated based on a captured image of the camera. The tone information included in the intermediate language data generated by the above-described aspect is generated based on the actual user utterance history. Therefore, it is expected that the tone information reproduces the wrinkles and the like when the user speaks.

  In addition, the code | symbol in the parenthesis described in the claim shows the correspondence with the specific means as described in embodiment mentioned later as one aspect, Comprising: The technical scope of this invention is limited is not.

1 is a block diagram showing a schematic configuration of a voice input system 100. FIG. It is a block diagram showing notionally the function with which learning processing part F8 is provided. It is a flowchart for demonstrating standby mode process. It is a flowchart for demonstrating a pattern learning process. It is a figure for demonstrating a pattern learning process. FIG. 6 is a diagram conceptually showing syllable set data. It is a flowchart for demonstrating speech content recognition processing. It is a flowchart for demonstrating intermediate language data generation processing.

  Hereinafter, a voice input system 100 to which the present invention is applied will be described with reference to the drawings. The voice input system 100 is mounted on a vehicle and includes a voice recognition device 1, a sheet sensor 2, a talk switch (hereinafter referred to as talk SW) 3, a microphone 4, and a camera 5, as shown in FIG. . Each of the sheet sensor 2, the talk SW 3, the microphone 4, and the camera 5 is configured to be able to communicate with the voice recognition device 1 via a local network (hereinafter, LAN: Local Area Network) built in the vehicle. .

  The speech recognition apparatus 1 is configured as a normal computer including a CPU, a RAM, a ROM, an I / O, and a bus line that connects these configurations. The ROM stores a program for causing a normal computer to function as the speech recognition apparatus 1 (hereinafter referred to as an utterance content specifying program) and the like.

  Note that the above-described utterance content identification program is not limited to the ROM, but may be stored in a non-transitionary tangible storage medium. The execution of the utterance content identification program by the CPU corresponds to the execution of a method corresponding to the utterance content identification program.

  In general, the voice recognition device 1 recognizes the content of a user's utterance based on data input from the microphone 4 or the camera 5 and uses the recognized result as predetermined application software (hereinafter referred to as application software). ). Details of the voice recognition device 1 will be described later. The user here refers to a person who uses a vehicle on which the voice input system 100 is mounted, and particularly refers to a person sitting in a driver's seat. The voice recognition device 1 corresponds to the utterance content recognition device described in the claims.

  The seat sensor 2 is a sensor that outputs a signal indicating whether an occupant (that is, a user) is seated in the driver's seat. The sheet sensor 2 can be realized using a pressure sensor, for example. That is, the seat sensor 2 is a pressure sensor provided in the seating portion of the driver's seat, and outputs a signal indicating the pressure acting on the seating portion to the voice recognition device 1.

  The talk SW3 is a switch for instructing the user to start voice input. As an example, the talk SW 3 is a so-called click-type switch. When the talk SW 3 is turned on by a user operation (ie, clicked), an on signal is output to the voice recognition device 1. The talk SW3 is provided at a position where the user can easily operate, for example, near the side surface of the steering column cover or the vicinity of the shift lever.

  The talk SW3 may be a button image displayed on the display. In that case, the speech recognition apparatus 1 may detect that the button image has been selected by the user via a touch panel or a known pointing device.

  The microphone 4 is, for example, a small omnidirectional microphone, collects surrounding sounds such as voices and noises spoken by the user, converts them into electrical voice signals, and outputs them to the voice recognition device 1. The microphone 4 is provided at a position where the user's voice can be easily picked up, such as the upper surface portion of the steering column cover or the sun visor on the driver's seat.

  The camera 5 is an optical camera, and for example, a CMOS camera or a CCD camera can be used. The camera 5 is disposed at an appropriately designed position such as a steering column cover or a portion of the instrument panel facing the driver's seat so as to photograph the face of the occupant seated in the driver's seat. . In addition, as another aspect, the camera 5 may be an infrared camera, a near infrared camera, or the like. The camera 5 sequentially outputs image data captured at a predetermined frame rate (for example, 30 fps) to the voice recognition apparatus 1. The camera 5 may output a captured image as a video signal. An image taken by the camera 5 corresponds to the user image described in the claims.

<About the voice recognition device 1>
The speech recognition apparatus 1 provides functions corresponding to various functional blocks shown in FIG. 1 when the CPU executes the above-described utterance content identification program. Specifically, the voice recognition device 1 includes a seating determination unit F1, an operation reception unit F2, a voice acquisition unit F3, a voice recognition unit F4, a noise level determination unit F5, an image acquisition unit F6, and a lip reading processing unit F7 as functional blocks. , A learning processing unit F8, an intermediate languageizing unit F9, and a recognition medium setting unit F10.

  Note that some or all of the functional blocks provided in the speech recognition apparatus 1 may be realized as hardware using one or a plurality of ICs. Further, it may be realized by a combination of execution of software by the CPU and hardware members.

  In addition, the speech recognition apparatus 1 includes a speech recognition DB 1a, a lip reading DB 1b, and a tone DB 1c as databases (hereinafter referred to as DB) realized using a nonvolatile storage medium. These DBs may be realized using a non-volatile rewritable storage medium such as a hard disk or a flash memory.

  The speech recognition DB 1a is a DB that stores data necessary for speech recognition processing. Data necessary for speech recognition processing includes, for example, an acoustic model in which acoustic features of small units of human speech (so-called phonemes) are described, a recognition dictionary that associates phonetic acoustic features with words, and concatenation between words A language model that expresses relationships.

  The lip reading DB 1b is a DB in which data necessary for lip reading processing described later is stored. The data required for the lip reading process is, for example, a lip movement model showing a lip shape change pattern (hereinafter referred to as lip movement pattern) for each syllable character, or a word dictionary describing words that the user has used in syllable characters. Etc. In the word dictionary, it is preferable that the usage frequency for each word and information about other words used before and after the word are associated with each other. Note that the syllable characters refer to kana such as hiragana and katakana.

  In the lip movement model, at least five Japanese vowels “A”, “I”, “U”, “E”, “O” are added to the syllable character “n” as a repellent sound. In addition, image data indicating lip movement patterns corresponding to six voice characters is registered. Of course, the lip movement model also includes image data indicating a lip movement pattern for a syllable formed by combining vowels and consonants. Note that syllables that are a combination of vowels and consonants are straight sounds such as “ka”, “ki”, “ku”, “ke”, and “ko”, and roaring sounds such as “kya”, “kyu”, and “kyo”. , “She”, “Chee”, etc. It is only necessary to appropriately design whether to introduce the concept of mora and to handle prompt sounds and long sounds as one syllable. Data indicating the lip movement pattern for each syllable character may be generated in the pattern learning process described later.

  The tone DB 1c is a DB that stores syllable set data including tone pattern data described later. The tone pattern data and syllable set data will be described later separately.

  The seating determination unit F1 determines whether or not the user is seated in the driver's seat based on a signal input from the seat sensor 2. Specifically, when the seat sensor 2 detects a pressure equal to or higher than a predetermined threshold (hereinafter referred to as a seating determination threshold), it is determined that the user is seated in the driver's seat. When the pressure detected by the seat sensor 2 is less than the seating determination threshold, it is determined that the user is not seated in the driver's seat.

  In the present embodiment, as an example, a mode in which it is determined whether or not the user is seated in the driver's seat using the seat sensor 2 is adopted, but this is not a limitation. As another aspect, it may be determined whether or not the user is seated in the driver's seat based on the captured image of the camera 5. In that case, if the user's face can be detected in the captured image, it may be determined that the user is seated in the driver's seat.

  The operation reception unit F2 detects an operation in which the user presses the talk SW3. That is, the operation reception unit F2 receives a user operation (hereinafter referred to as a voice input operation) for the user to start voice input.

  By the way, the speech recognition apparatus 1 includes three operation modes, which are a sleep mode, a standby mode, and a recognition execution mode, as operation modes. Switching of the operation mode is performed by an operating system (hereinafter referred to as OS) (not shown).

  The sleep mode is an operation mode when the voice recognition device 1 is activated and no occupant is seated in the driver's seat. In the sleep mode, the voice recognition device 1 turns off the microphone 4 and the camera 5.

  When the user's seating in the driver's seat is detected in the sleep mode, the microphone 4 and the camera 5 are turned on and the standby mode is entered. It is assumed that the voice recognition device 1 is configured to be activated when a vehicle power source (for example, an ignition power source) is turned on or when a vehicle door is opened.

  The standby mode is an operation mode when the user is not seated in the driver's seat and the voice input operation by the user is not executed (in other words, in the input standby state). When the operation receiving unit F2 receives a voice input operation while the voice recognition device 1 is operating in the standby mode, the voice recognition device 1 shifts from the standby mode to the recognition execution mode to recognize the user's utterance content. Processing (hereinafter referred to as speech content recognition processing). When the processing is completed, the process returns to the standby mode. In other words, the state in which the utterance content recognition process is executed corresponds to the recognition execution mode. Note that when the user leaves the driver's seat in the standby mode, the mode shifts to the sleep mode. The utterance content recognition process will be described later separately.

  The voice acquisition unit F3 acquires a voice signal output from the microphone 4. The voice acquisition unit F3 determines whether the user is speaking based on the input signal from the microphone 4. Whether or not the user is speaking can be determined based on, for example, the number of zero crossings of the audio signal. That is, when a signal exceeding a certain level is input and the number of zero crossings per predetermined unit time exceeds a certain number, the time when the utterance is started (hereinafter referred to as the utterance start point) is adopted. Also, the time when the condition is no longer satisfied is adopted as the time when the utterance ends (hereinafter referred to as the utterance end point). Hereinafter, the period from the utterance start point to the utterance end point is referred to as an utterance interval, and a period other than the utterance interval is referred to as a non-utterance interval.

  Further, the voice acquisition unit F3 performs A / D conversion on the voice signal input during the period determined to be the utterance section, thereby digital data corresponding to the voice uttered by the user (hereinafter referred to as utterance). Audio data). The utterance voice data is stored in a memory (not shown) in association with time information including the utterance start time.

  It should be noted that the speech segment and the non-speech segment may be identified by other known methods. For example, a method of detecting the start and end of an utterance interval based on a Gaussian Mixture Model (GMM) may be employed. This defines each GMM for speech and non-speech, calculates the likelihood of each GMM from feature extraction for each input short frame, and starts the speech segment from the likelihood ratio of speech GMM and non-speech GMM. This is a method for determining the end.

  The speech recognition unit F4 performs speech recognition processing on the utterance speech data generated by the speech acquisition unit F3, using various data stored in the speech recognition DB 1a. Since the voice recognition process may use a known technique, a description thereof is omitted here.

  The noise level determination unit F5 determines the noise level (that is, the noise level) based on the amplitude of the audio signal input from the microphone 4 when it is determined by the audio acquisition unit F3 that it is a non-speech segment. For example, the noise level is determined to be high when the amplitude of the audio signal input in the non-speech section exceeds a predetermined threshold value, and the noise level is determined to be less than the threshold value. The level is determined to be low.

  The image acquisition unit F6 sequentially acquires image data captured by the camera 5. The image acquisition unit F6 adds a time stamp indicating the acquisition time to the acquired image data, and stores it in a memory (not shown). When the capacity of the image data stored in the memory reaches a predetermined upper limit, it is only necessary that the acquisition time is deleted sequentially from the oldest data.

  The lip reading processing unit F7 detects the movement of the user's lip from the image data acquired from the image acquisition unit F6. Then, the lip reading processing unit F7 specifies the time when the user starts utterance (that is, the utterance start point) and the time when the utterance ends (that is, the utterance end point) from the presence or absence of the movement of the user's lips. That is, the utterance section is specified.

  Also, the voice uttered by the user is converted into text from the change pattern (that is, lip movement pattern) of the user's lip shape in a series of image data (hereinafter referred to as utterance image data) captured in the utterance section. That is, the lip reading processing unit F7 executes lip reading processing. Note that the textification here is to generate a string of syllable characters (hereinafter referred to as an utterance character string) corresponding to the user's utterance voice.

  The generation of the utterance character string may be realized by comparing the lip movement pattern specified from the image data with the lip movement pattern for each syllable character stored as the lip movement model in the lip reading DB 1b. A known method such as dynamic programming can be used as the comparison method. When a plurality of syllable characters are extracted as candidates for the lip movement pattern for one character, word candidates spoken by the user are extracted using syllable characters uttered before and after that, and the word dictionary is referred to Thus, characters that form words with high likelihood may be employed. In addition, a known method can be used as an algorithm for generating an utterance character string from image data.

  The learning processing unit F8 is a functional block that executes a pattern learning process to be described later. As shown in FIG. 2, the learning processing unit F8 has finer functions (ie, sub-functions) for performing the pattern learning processing, as shown in FIG. 2, a tone pattern specifying unit F81, a connected lip movement pattern specifying unit F82, and a storage processing unit F83. Is provided. Details of these sub-functions and pattern learning processing will be described later.

  The intermediate language unit F9 performs intermediate language data generation processing to be described later. The intermediate language unit F9 sweeps away the intermediate language data generation unit described in the claims. Based on the determination result of the noise level determination unit F5, the recognition medium setting unit F10 switches between using the voice recognition unit F4 and the lip reading processing unit F7 to specify the content of the user's utterance. Specifically, when the noise level is determined to be low, the voice recognition unit F4 is set as a means for identifying the user's utterance content (hereinafter, recognition medium), while the noise level is high. If it has been determined, the lip reading processing unit F7 is set as a recognition medium.

<Standby mode processing>
Next, processing (hereinafter referred to as standby mode processing) executed when the speech recognition apparatus 1 is operating in the standby mode will be described using the flowchart shown in FIG. The flowchart shown in FIG. 3 is started when it is detected that the user sitting on the driver's seat is seated. It is also started when an utterance content recognition process described later is completed. That is, it may be started when a transition is made from the sleep mode or the recognition execution mode to the standby mode.

  In addition, when shifting from the sleep mode to the standby mode, the camera 5 and the microphone 4 are turned on. Independently of the standby mode, the noise level determination unit F5 is assumed to sequentially perform noise level determination based on the audio signal input from the microphone 4.

  First, in step S101, the voice acquisition unit F3 determines whether the user is speaking based on the voice signal input from the microphone 4. If it is determined that the user is speaking, an affirmative determination is made in step S101 and the process proceeds to step S102. On the other hand, if it is determined that the user is not speaking, a negative determination is made in step S101 and the process proceeds to step S104.

  In step S102, the voice acquisition unit F3 acquires utterance voice data, and the lip reading processing unit F7 extracts the utterance image data from the image data stored in the memory, and proceeds to step S103. In step S103, the learning processing unit F8 executes a pattern learning process and proceeds to step S104. The pattern learning process performed in step S103 will be described later separately.

  In step S <b> 104, the seating determination unit F <b> 1 determines whether the user is seated in the driver's seat based on the signal input from the seat sensor 2. If a signal indicating that the user is seated in the driver's seat is input from the seat sensor 2, an affirmative determination is made in step S104, and the process proceeds to step S105. On the other hand, when a signal indicating that the user is not seated in the driver's seat is input from the seat sensor 2, a negative determination is made in step S104, and the process proceeds to step S107.

  In step S105, the operation reception unit F2 determines whether or not the talk SW3 has been pressed. If the talk SW3 has been pressed, an affirmative decision is made in step S105 and the process proceeds to step S106. On the other hand, if the talk SW3 has not been pressed, a negative determination is made in step S105, and the process returns to step S101.

  In step S106, the operation mode is set to the recognition execution mode, and this flow ends. In step S107, the operation mode is set to the sleep mode, and this flow ends.

<Pattern learning process>
Next, the pattern learning process performed by the learning processing unit F8 will be described using the flowchart shown in FIG. This flowchart may be started when the process proceeds to step S103 of the standby mode process shown in FIG.

  First, in step S201, the speech recognition unit F4 performs speech recognition processing using the utterance speech data provided from the speech acquisition unit F3. By executing step S201, a character string (that is, an uttered character string) corresponding to the user's uttered voice is generated, and the timing at which each syllable character is uttered is specified. Further, as preparation for subsequent processing, numbers (hereinafter referred to as utterance numbers) are assigned to syllable characters uttered by the user in the order of utterance. When step S201 is completed, the process proceeds to step S202.

  In step S202, the lip reading processing unit F7 detects the movement of the user's lip from the utterance image data, and sequentially identifies the frame portion in which the user utters each syllable character in the series of utterance image data. Then, the lip movement pattern when uttering each syllable character is specified. When the process of step S202 is completed, the process proceeds to step S203. In step S203, the learning processing unit F8 stores the lip movement pattern when each syllable character is uttered in the lip reading DB 1b.

  By performing such steps S201 to S203, the speech recognition apparatus 1 learns the user's lip movement pattern for each syllable character. FIG. 5 conceptually shows the flow of processing from steps S201 to S203.

  The speech recognition unit F4 sequentially identifies syllable characters uttered by the user as shown in FIG. 5B by performing speech recognition processing on the speech voice data shown in FIG. Go. That is, an utterance character string is generated. Also, utterance numbers are assigned in the order in which each syllable character is uttered. Then, the learning processing unit F8 registers the image data corresponding to each state in the lip reading DB 1b as the lip movement pattern of the syllable characters uttered at that time. Note that (C) in FIG. 5 represents the utterance number assigned to each syllable character, and (D) represents the lip movement pattern corresponding to each syllable character.

  Hereinafter, the state in which the j-th (j is an integer) syllable character from the beginning in a series of uttered character strings is also referred to as the j-th state. Further, the state immediately before the first voice (that is, the state of no voice) is handled as the zeroth state. In addition, a single utterance number is assigned to an unvoiced state immediately after the end of utterance. In FIG. 5, the non-speech state immediately after the end of the utterance is set to the eighth state.

  Returning to FIG. 4 again, the description of the pattern learning process is continued. In step S204, the variable j used for the subsequent processing is set to 1, and the process proceeds to step S205. In step S205, the number n of syllable characters constituting the utterance character string is acquired, and the process proceeds to step S206. n is a natural number. In the example shown in FIG. 5, n = 7.

  In step S206, it is determined whether j is less than n + 1. When j is less than n + 1, step S206 is affirmed and the process proceeds to step S207. On the other hand, if j is greater than or equal to n + 1, a negative determination is made in step S206, and this flow ends. If this flow is completed, the process returns to the standby mode process that is the caller of this flow, and the process proceeds to step S104.

  In step S207, the tone pattern specifying unit F81 continuously utters the j-1st syllable and the jth syllable based on the speech data corresponding to the j-1st to jth states. (Hereinafter referred to as tone pattern data). That is, the tone pattern data is data indicating the tone for two syllables, the tone of the first syllable and the tone of the second syllable.

  The tone here includes various parameters such as intonation (so-called intonation), the pitch of the user's voice, speaking speed, volume, and the like. 6A and 6B conceptually represent tone data when uttering “Kyo” and “U” in succession. Specifically, (A) represents a change in tone, and (B) represents a change in volume. Although the illustration of the speaking speed is omitted, the speaking speed may be quantified by a well-known method. Note that the types of items constituting the tone data may be designed as appropriate. It is assumed that the intonation and the pitch of the voice are expressed by data indicating the tone.

  Various well-known formats can be adopted as the representation format of the tone data. Here, as an example, it is expressed in a format defined as an ITS on-board unit speech synthesis symbol (JEITA TT-6004) in the standards of the Japan Electronics and Information Technology Industries Association.

  If j = 1, that is, if the j-1 state is an unvoiced state, step S207 generates tone pattern data for uttering the first syllable character from the unvoiced state. It corresponds to processing. When the process in step S207 is completed, the process proceeds to step S208.

  In step S208, when the connected lip movement pattern specifying unit F82 continuously utters the j-1 syllable and the jth syllable based on the utterance image data corresponding to the j-1st to jth states. Lip movement pattern (hereinafter referred to as connected lip movement pattern) is specified. Then, connected lip movement data indicating the connected lip movement pattern is generated. FIG. 6C conceptually shows the connected lip movement pattern when “Kyo” and “U” are continuously spoken. When the process in step S208 is completed, the process proceeds to step S209.

  In step S209, the storage processing unit F83 stores the tone data generated in step S207, the connected lip movement data generated in step S208, and the two syllable characters indicated by them in the tone DB 1c. For convenience, the data that associates the tone data generated in step S207 with the connected lip movement data generated in step S208 is referred to as syllable set data. When the process in step S209 is completed, the process proceeds to step S210.

  In step S210, the value of variable j is incremented by 1 (that is, incremented), and the process returns to step S206. Therefore, by repeating steps S206 to S210, syllable set data for every two consecutive syllables is generated. For example, in the example shown in FIG. 5, seven syllable set data are generated.

  In the tone DB 1c, various syllable set data is stored by grouping, for example, two syllable characters indicated by the syllable set data as labels. When there are a plurality of syllable set data for the utterance “Kyo”, they are grouped and stored as data corresponding to two syllables “Kyo”. The case where there are a plurality of syllable set data for the utterance “Kyo” corresponds to the case where the user has uttered “Kyo” in various tone patterns or connected lip movement patterns in the past.

<Speech content recognition processing>
Next, the speech content recognition process performed by the learning processing unit F8 will be described using the flowchart shown in FIG. The utterance content recognition process is a process for identifying the content uttered by the user based on the voice collected by the microphone 4 or the captured image of the camera 5 (in other words, using either voice recognition or lip reading process). is there. The utterance content recognition process may be started when the talk SW 3 is pressed. That is, it starts when the operation mode shifts to the recognition execution mode.

  First, in step S301, the recognition medium setting unit F10 determines, based on the determination result of the noise level determination unit F5, which of the speech recognition unit F4 and the lip reading processing unit F7 is used to specify the user's utterance content. If it is determined that the noise level is low, it is determined to use the voice recognition unit F4 to specify the user's utterance content, and the process proceeds to step S310. On the other hand, if the noise level is determined to be high, it is determined that the utterance content of the user is specified using the lip reading processing unit F7, and the process proceeds to step S320.

  In step S310, the voice recognition unit F4 acquires the speech voice data generated by the voice acquisition unit F3, and proceeds to step S311. In step S311, the voice recognition unit F4 performs voice recognition processing based on the acquired utterance voice data, and proceeds to step S330.

  In step S320, utterance image data is acquired, and the process proceeds to step S321. In step S321, the lip reading processing unit F7 performs a lip reading process using the utterance image data acquired in step S321, thereby generating an utterance character string, and proceeds to step S322.

  In step S322, the intermediate language unit F9 performs intermediate language data generation processing using the utterance character string generated in step S321, and proceeds to step S323. The intermediate language data generation process will be described later separately. As a product of the intermediate language data generation process, data (hereinafter referred to as intermediate language data) is generated by adding tone information indicating the tone when the user utters each syllable character to the utterance character string.

  In step S323, the speech recognition unit F4 specifies the content of the user's utterance by executing speech recognition processing using the intermediate language data generated in step S323. The specification of the utterance content here is, for example, to divide the utterance character string into word levels based on intonation, and further convert it into a single sentence with meaning based on the concatenated relationship between words. When the process in step S323 is completed, the process proceeds to step S330.

  In step S330, data indicating the utterance content of the user specified by the above processing is provided to a predetermined application, and the process proceeds to step S331. In step S331, the operation mode is shifted to the standby mode and this flow is terminated. When this flow is finished, the standby mode process shown in FIG. 3 is started.

<Intermediate language data generation processing>
Next, intermediate language data generation processing performed by the intermediate language unit F9 will be described using the flowchart shown in FIG. This flowchart may be started when the process proceeds to step S322 of the utterance content recognition process shown in FIG.

  First, in step S401, the number n of syllable characters included in the utterance character string generated by the lip reading processing unit F7 is acquired, and the process proceeds to step S402. In step S402, the variable k used for the subsequent processing is set to 1, and the process proceeds to step S403. Note that k is a variable in which a natural number is set.

  In step S403, it is determined whether k is less than n + 1. When k is less than n + 1, step S403 is affirmed and the process proceeds to step S404. On the other hand, if k is greater than or equal to n + 1, a negative determination is made in step S403 and this flow ends. When this flow is completed, the process returns to the utterance content recognition process that is the caller of this flow, and the process proceeds to step S323.

  In step S404, the lip movement pattern (hereinafter referred to as observation) when the k-1th syllable and the kth syllable are continuously spoken based on the utterance image data corresponding to the k-1st to kth states. Lip movement pattern). When the process in step S404 is completed, the process proceeds to step S405. The kth syllable character corresponds to the target character recited in the claims, and the (k-1) th syllable character corresponds to the syllable character positioned immediately before the target character recited in the claims.

  The observed lip movement pattern may be specified by the intermediate language conversion unit F9 or the connected lip movement pattern specification unit F82. Further, the intermediate language conversion unit F9 may specify the lip movement pattern specified by the lip reading processing unit F7 when generating the utterance character string. In any case, the observed lip movement pattern is specified based on the image data used to generate the utterance character string.

  In step S405, the connected lip movement data indicating the connected lip movement pattern having the highest degree of similarity to the observed lip movement pattern specified in step S404 is specified from among the various connected lip movement data stored in the tone DB 1c. . Here, as an example, connected lip movement data having the k-1th syllable and the kth syllable character as labels is extracted, and among them, the connected lip movement pattern having the highest degree of similarity to the observed lip movement pattern. It is assumed that the connected lip movement data indicating is selected.

  The degree of similarity may be calculated using a known method such as pattern matching. If only one connected lip movement data assigned with the k-1th syllable and the kth syllable character as labels is registered in the tone DB 1c, the connected lip movement data is selected. Good. When the process in step S405 is completed, the process proceeds to step S406.

  In step S406, the tone data associated with the connected lip movement data selected in step S405 is read, and the process proceeds to step S407. In step S407, the tone for the kth syllable character is determined from the read tone data and the tone assigned to the k-1th syllable character. For example, when k = 1, the second syllable tone shown in the read tone data is used as it is.

  Also, when k ≧ 2, the read tone data is set so that the tone of the first syllable indicated in the read tone data matches the tone set for the k−1 syllable character in the utterance character string. Correct the tones for the two syllables shown in. For example, when the tone of the first syllable indicated in the read tone data is 0.5 octaves lower than the tone set for the k-1 syllable character in the utterance character string, the two tones shown in the tone data Raise the tone for both syllables by 0.5 octaves. Then, the tone of the second syllable of the tone data subjected to such correction is adopted as the tone for the kth syllable character.

  When the process in step S407 is completed, the process proceeds to step S408. In step S408, the value of the variable k is increased by 1 (that is, incremented), and the process returns to step S403. Therefore, the tone for all the syllable characters constituting the utterance character string is determined by repeating steps S403 to S408. That is, intermediate language data in which tone information is added to the utterance character string is generated. As the expression format of the intermediate language data, any format such as JEITA TT-6004 can be adopted as described above.

<Summary of Embodiment>
In the above configuration, when the user's utterance is detected while operating in the standby mode, the connected lip movement data, the tone pattern data, and the voice data uttered by the user and the image data at that time, Is stored in the tone DB 1c (step S103).

  When the lip reading process is performed with the user's press of the talk SW 3 as a trigger, the connected lip movement pattern is specified from the image data at the time of the user's utterance, and is stored in the specified connected lip movement pattern and tone DB 1c. The tone of each syllable is determined from the data being recorded.

  That is, according to the above configuration, intermediate language data can be generated from image data captured by the camera 5. Further, since the tone assigned to each syllable is determined based on the lip movement pattern and tone pattern when the user actually speaks, it is expected that the tone is close to the tone of the actual user. Therefore, the intermediate language data generated by the above-described method can be expected to be intermediate language data that reproduces the user's tone relatively accurately.

  Note that, in general, intermediate language data to which tone information when a utterance is added is larger in amount of information than a simple sequence of syllable characters (that is, an utterance character string). Therefore, when analyzing the utterance content, the use of intermediate language data rather than the utterance character string improves the accuracy of identification such as whether a word breaks or whether it is a question sentence, and provides a more appropriate recognition result. It becomes like this. That is, according to the above configuration, the utterance content can be recognized more accurately based on the result of the lip reading process.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the above-mentioned embodiment, The various modifications described below are also contained in the technical scope of this invention, and also in addition to the following However, various modifications can be made without departing from the scope of the invention.

  In addition, about the member which has the same function as the member described in the above-mentioned embodiment, the same code | symbol is attached | subjected and the description is abbreviate | omitted. In addition, when only a part of the configuration is mentioned, the configuration of the above-described embodiment can be applied to the other portions.

[Modification 1]
In the above, although the aspect which utilizes the produced | generated intermediate language data for the specification (in other words recognition) of utterance content was disclosed, it is not restricted to this. The intermediate language data may be used for speech synthesis processing. In that case, the speech recognition apparatus 1 provides the intermediate language data generated by the intermediate language unit F9 to application software that executes speech synthesis processing.

[Modification 2]
In the above-described embodiment, the mode in which the microphone 4 is turned off when the user leaves the seat is disclosed, but this is not a limitation. The microphone 4 may always be kept on while the traveling power supply is on.

[Modification 3]
When there are a plurality of persons (that is, users) who use the vehicle, the various processes described above are preferably performed by identifying the user. That is, it is preferable that a user is identified by a face image, a voice print, a fingerprint, etc., and a lip movement pattern, linked lip movement data, and tone data for each syllable character is generated for each user.

[Modification 4]
In the above, although the aspect which processes by uttering a user's speech by the concept of a syllable is disclosed, it is not limited to this. The user's speech may be divided and processed by the concept of mora.

100 voice input system, 1 voice recognition device, 2 sheet sensor, 3 talk switch, 4 microphone, 5 camera, F1 seating determination unit, F2 operation reception unit, F3 voice acquisition unit, F4 voice recognition unit, F5 noise level determination unit, F6 image acquisition unit, F7 lip reading processing unit, F8 learning processing unit, F9 intermediate languageization unit (intermediate language data generation unit), F10 recognition medium setting unit, F81 tone pattern specifying unit, F82 connected lip movement pattern specifying unit, F83 storage Processing unit, 1a database for speech recognition, 1b database for lip reading, 1c tone database

Claims (4)

An utterance content recognition device that executes a process of specifying the utterance content of a user with a predetermined user operation as a trigger,
An operation reception unit (F2) for receiving the user operation;
A voice acquisition unit (F3) that acquires the user's speech voice as speech voice data via a microphone;
An image acquisition unit (F6) for sequentially acquiring user images, which are images captured by a camera arranged to capture the user's face;
A lip movement pattern that is a change pattern of the user's lip shape is detected from the user image acquired by the image acquisition unit, and further, a character string corresponding to the user's utterance voice based on the detected lip movement pattern. A lip reading processing unit (F7) for generating a certain utterance character string;
A tone pattern specifying unit (F81) for specifying a tone pattern when two syllables are uttered continuously based on the utterance voice data acquired by the voice acquisition unit;
A connected lip movement pattern specifying unit (F82) for specifying a connected lip movement pattern which is a lip shape change pattern when the user continuously utters two syllables from the user image acquired by the image acquisition unit; ,
For the two syllables continuously uttered by the user, the connected lip movement pattern specified by the connected lip movement pattern specifying unit and the tone pattern specified by the tone pattern specifying unit are stored in a tone database in association with each other. A learning processing unit (F8) that executes pattern learning processing as processing,
Intermediate language data in which tone information is added to each syllable character constituting the utterance character string using the data stored in the tone database and the user image used to generate the utterance character string An intermediate language data generation unit (F9) for generating
The learning processing unit sequentially executes the pattern learning process in cooperation with the tone pattern specifying unit and the connected lip movement pattern specifying unit when the operation receiving unit does not receive the user operation,
The intermediate language data generation unit generates the intermediate language data when the lip reading processing unit generates the utterance character string based on the operation reception unit receiving the user operation,
The intermediate language data generation unit
When determining the tone of a target character that is a certain syllable character constituting the utterance character string, the target character is selected from a plurality of the connected lip movement patterns stored in the tone database. Identifying the connected lip movement pattern having a high similarity to the change pattern of the user's lip shape when the syllable character positioned immediately before and the target character are uttered;
An utterance content recognition apparatus, wherein a tone of the target character is determined using the tone pattern associated with the identified connected lip movement pattern. In claim 1,
A speech recognition unit (F4) that performs speech recognition processing based on the utterance speech data acquired by the speech acquisition unit;
A noise level determination unit (F5) that determines a noise level that is a noise level based on the amplitude of the audio signal output from the microphone;
When the noise level determination unit determines that the noise level is high when the operation reception unit receives the user operation, the lip reading processing unit converts the uttered character string based on the user image. While generating
When the noise level is determined to be low by the noise level determination unit at the time when the operation reception unit receives the user operation, the voice recognition unit executes the voice recognition process so that the user An utterance content recognition device characterized by specifying the utterance content. In claim 2,
The speech recognition unit is characterized in that the speech recognition unit specifies the speech content of the user by performing the speech recognition process using the intermediate language data generated by the intermediate language data generation unit. In any one of Claims 1-3,
The utterance content recognition apparatus, wherein the intermediate language data generated by the intermediate language data generation unit is provided to application software for executing speech synthesis processing.

Images