JP2003271181A - Information processor, information processing method, recording medium and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To register an unregistered word that has not been registered in a dictionary by more accurately clustering the unregistered word. <P>SOLUTION: A word dictionary stored in a dictionary storage part 25 stores the RGB values of an image corresponding to a word in addition to the headline of the word and a phoneme sequence. For example, RGB values representing red are stored in a word 'red'. A matching part 23 refers to the word dictionary to perform matching processing on the basis of both a speech feature vector extracted by a feature extraction part 22 and the feature vector of an image extracted by a feature extraction part 42. When, for example, the speech of 'a red ball' is inputted, the 'red' part is subjected to recognition processing by utilizing the speech and also the feature as an image. This invention can be applied to a speech recognition device. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing apparatus and an information processing method, a recording medium and a program, and in particular, for example, a dictionary of a word or the like which is a target of speech recognition can be simply and accurately defined. The present invention relates to an information processing device and an information processing method, a recording medium and a program that can be registered in.

[0002]

2. Description of the Related Art For example, in order for a robot to understand the speech of a user (human), it is necessary to equip the robot with a voice recognition device. Usually, in a voice recognition device, a dictionary is prepared and words are registered in advance in the dictionary. When a voice is input, it is determined whether or not the voice includes a word (registered word) registered in the dictionary. If the voice is included, the registered word is output as a voice recognition result. .

As described above, since the voice recognition device recognizes the input voice by referring to the dictionary, basically, the words other than the registered words (unregistered words) are recognized.
Will not be able to recognize.

As one of the typical measures against such unregistered words, when the input voice includes unregistered words, the unregistered words are registered in the dictionary, and thereafter, There is a method to use it as a registered word.

To register an unregistered word in the dictionary, it is first necessary to detect the voice section of the unregistered word and recognize the phoneme sequence of the voice in the voice section. As a method of recognizing a phoneme sequence of a certain speech, for example, there is a method called a phoneme typewriter. In the phoneme typewriter, basically, a garbage model that allows free transitions for all phonemes is used to input speech. The phonological sequence for is output.

Further, to register an unregistered word in the dictionary,
It is necessary to cluster the phoneme sequences of unregistered words.
That is, in the dictionary, the phonological sequence of each word is registered by being clustered into the cluster of that word. To register an unregistered word in the dictionary, it is necessary to cluster the phonological sequence of that unregistered word. .

[0007]

As a method of clustering a phoneme sequence of unregistered words, a user is requested to input a heading representing the unregistered word (for example, reading of the unregistered word), and the heading is used. There is a method of clustering the phoneme sequence of unregistered words in the represented cluster, but this method is troublesome because the user has to input the headline.

There is also a method of generating a new cluster each time an unregistered word is detected and clustering a phoneme sequence of the unregistered word into the new cluster. However, in this method, each time an unregistered word is detected, an entry corresponding to a new cluster is registered in the dictionary, so that the dictionary becomes large in scale and the processing amount required for subsequent speech recognition is increased. And time will increase.

Further, when an unregistered word is registered, the unregistered word is processed only based on voice input. Therefore, for example, when the speaker has a dullness,
Even if they are the same word, they will be registered as different words, or conversely, even if they are different words,
Sometimes it was registered as the same word.

Further, in order for the robot to recognize colors, it is necessary for the robot to learn colors. In such a case, conventionally, for example, specific values of three primary colors (RGB) representing colors captured by a video camera are registered. Then, the color included in the image captured by the video camera is compared with the color registered in advance to generally determine the color of the currently captured image.

However, for example, since red and orange are relatively similar colors, red imaged by a video camera may be erroneously recognized as orange, or orange may be erroneously recognized as red. It was

The present invention has been made in view of such a situation, and the input information is accurately and easily clustered without increasing the registered data amount, and thus the It is intended to be able to recognize.

[0013]

An information processing apparatus of the present invention is an information processing apparatus for recognizing input information based on registration information, and first acquisition means for acquiring first information,
Second acquisition means for acquiring the second information, first feature extraction means for extracting the characteristics of the first information acquired by the first acquisition means, and second acquisition means for the second acquisition means Two
The second characteristic extracting means for extracting the characteristic of the information, the characteristic of the first information extracted by the first characteristic extracting means, and the characteristic of the second information extracted by the second characteristic extracting means. Using, based on the clustering means for clustering the first information acquired by the first acquisition means, and the result of clustering by the clustering means,
And a registration unit for registering the first information as registration information.

The clustering means obtains the weighted addition of the characteristics of the first information extracted by the first characteristic extraction means and the characteristics of the second information extracted by the second characteristic extraction means. The first information may be clustered based on the identified features.

The first information and the second information may be one or the other of voice and image.

The first or second feature extracting means can extract the features of the image as RGB values.

The first information may be a voice of an unregistered word that is not registered as registration information.

An information processing method of the present invention is an information processing method of an information processing apparatus for recognizing input information based on registration information, which comprises a first acquisition step of acquiring first information,
By the second acquisition step of acquiring the second information, the first feature extraction step of extracting the characteristics of the first information acquired by the processing of the first acquisition step, and the processing of the second acquisition step A second characteristic extraction step for extracting the characteristic of the acquired second information, a characteristic of the first information extracted by the processing of the first characteristic extraction step, and an extraction by the processing of the second characteristic extraction step Based on the clustering step of clustering the first information acquired by the processing of the first acquisition step using the characteristics of the generated second information and the result of the clustering performed by the processing of the clustering step, And a registration step of registering the information as registration information.

A program of a recording medium of the present invention is a program of an information processing apparatus for recognizing input information based on registration information, and includes a first acquisition step of acquiring first information and second information. A second acquisition step of acquiring, a first characteristic extraction step of extracting a characteristic of the first information acquired by the processing of the first acquisition step, and a second characteristic acquired by the processing of the second acquisition step A second feature extracting step for extracting the feature of the information of the first information, a feature of the first information extracted by the process of the first feature extracting step, and a second feature
Clustering step of clustering the first information acquired by the processing of the first acquisition step, using the characteristics of the second information extracted by the processing of the characteristic extraction step, and clustering by the processing of the clustering step. A registration step of registering the first information as registration information based on the result.

The program of the present invention is a program that can be executed by a computer that controls an information processing apparatus that recognizes input information based on registration information, and includes a first acquisition step for acquiring first information, and a first acquisition step for acquiring first information. Second acquisition step for acquiring the second information, a first characteristic extraction step for extracting the characteristic of the first information acquired by the processing of the first acquisition step, and a second characteristic acquisition step for the processing of the second acquisition step A second feature extracting step for extracting the feature of the extracted second information, a feature of the first information extracted by the process of the first feature extracting step, and a feature of the second feature extracting step. The clustering step of clustering the first information acquired by the processing of the first acquisition step by using the characteristics of the second information and the processing of the clustering step. Based on the clustered results, characterized by comprising a registration step of registering the first information as registration information.

In the information processing apparatus and the information processing method, the recording medium and the program of the present invention, the first
The first information is clustered using the information feature of and the second information feature. Then, based on the result of clustering, the first information is registered as registration information.

[0022]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an external configuration example of an embodiment of a robot to which the present invention is applied, and FIG.
An example of its electrical configuration is shown.

In the present embodiment, the robot 1 is in the shape of a four-legged animal such as a dog, and the leg units 3 are provided to the front, rear, left and right of the body unit 2, respectively.
A, 3B, 3C and 3D are connected, and the head unit 4 is provided at the front end and the rear end of the body unit 2, respectively.
And the tail unit 5 are connected to each other.

The tail unit 5 is drawn out from the base portion 5B provided on the upper surface of the body unit 2 so as to be curved or swingable with two degrees of freedom.

The body unit 2 contains a controller 10 for controlling the entire robot 1, a battery 11 as a power source of the robot 1, an internal sensor unit 14 including a battery sensor 12 and a heat sensor 13, and the like. There is.

The head unit 4 includes a microphone (microphone) 15 corresponding to "ears" and a CCD corresponding to "eyes".
(Charge Coupled Device), CMOS (Complementary Metal
Oxide Semiconductor) video camera 16,
A touch sensor 17 corresponding to a tactile sense, a speaker 18 corresponding to a “mouth”, and the like are provided at predetermined positions. Further, a lower jaw 4A corresponding to the lower jaw of the mouth is movably attached to the head unit 4 with one degree of freedom, and the opening / closing operation of the mouth of the robot 1 is realized by moving the lower jaw 4A. It is like this.

Joint parts of the leg units 3A to 3D, connecting parts of the leg units 3A to 3D and the body unit 2, connecting parts of the head unit 4 and the body unit 2, head unit 4 and the lower jaw linking moiety parts 4A, and the like in the connecting portion of the tail unit 5 and the body unit 2, as shown in FIG. 2, the actuators 3AA ₁ to 3AA _K, respectively, 3BA ₁ to 3BA _K, 3
CA ₁ to 3CA _K, 3DA ₁ to 3DA _K, 4A ₁ to 4
A _L , 5A ₁ and 5A ₂ are provided.

The microphone 15 in the head unit 4 collects ambient voice (sound) including the utterance from the user and sends the obtained voice signal to the controller 10. The video camera 16 captures an image of the surroundings and sends the obtained image signal to the controller 10.

The touch sensor 17 is provided, for example, on the upper part of the head unit 4, and is "stroked" by the user.
The pressure received by a physical action such as "tap" is detected, and the detection result is sent to the controller 10 as a pressure detection signal.

The battery sensor 12 in the body unit 2 detects the remaining amount of the battery 11 and sends the detection result to the controller 10 as a battery remaining amount detection signal. The heat sensor 13 detects heat inside the robot 1, and the detection result is used as a heat detection signal in the controller 1
Send to 0.

The controller 10 is a CPU (Central Pro
cessing unit) 10A, a memory 10B, and the like, and various processes are performed by the CPU 10A executing the control program stored in the memory 10B.

That is, the controller 10 includes a microphone 15,
Based on an audio signal, an image signal, a pressure detection signal, a battery remaining amount detection signal, or a heat detection signal given from the video camera 16, the touch sensor 17, the battery sensor 12, or the heat sensor 13, the surrounding situation or the user Judgment of whether or not there is a command from the user or the user's work.

Further, the controller 10 determines the subsequent action based on the result of the determination, and based on the result of the determination, the actuators 3AA _{1 to} 3AA _K , 3BA ₁
To 3BA _K , 3CA _{1 to} 3CA _K , 3DA _{1 to} 3DA
_K, 4A ₁ to 4A _L, 5A _1, 5A to drive the necessary of the _two. As a result, the head unit 4 is shaken vertically and horizontally, and the lower jaw 4A is opened and closed. Furthermore, the tail unit 5 is moved, and the leg units 3A to 3D are driven to cause the robot 1 to walk.

Further, the controller 10 generates a synthetic sound as needed, supplies it to the speaker 18 and outputs it, and an LED (Light Emitting Diode) (not shown) provided at the position of the "eyes" of the robot. Turn on, turn off or blink.

As described above, the robot 1 is adapted to act autonomously based on the surrounding situation and the like.

FIG. 3 shows an example of the functional configuration of the controller 10 shown in FIG. The functional configuration shown in FIG.
The CPU 10A is realized by executing the control program stored in the memory 10B.

The controller 10 accumulates the recognition results of the sensor input processing unit 50, the sensor input processing unit 50 that recognizes a specific external state, and the model storage unit 51 that expresses states such as emotion, instinct, and growth, and the sensor. An action determination mechanism unit 52 that determines a subsequent action based on the recognition result of the input processing unit 50, and a posture transition mechanism unit 53 that actually causes the robot 1 to make an action based on the determination result of the action determination mechanism unit 52,
Each actuator 3AA _{1 to} 3DA _K , 5A ₁ and 5
It is composed of a control mechanism unit 54 for driving and controlling A _2, and a voice synthesizing unit 55 for generating a synthetic sound.

The sensor input processing section 50 includes a microphone 15 and
Based on the audio signal, image signal, pressure detection signal, etc. provided from the video camera 16, the touch sensor 17, etc., a specific external state, a specific action from the user, an instruction from the user, etc. are recognized, and the recognition result The model storage unit 51 and the action determination mechanism unit 52 are notified of the state recognition information indicating the.

That is, the sensor input processing section 50 has a voice recognition section 50A, and the voice recognition section 50A is a microphone 1.
Voice recognition is performed on the voice signal given from 5. Then, the voice recognition unit 50A uses, as the state recognition information, a command such as "walk", "prone", "follow the red ball" as the voice recognition result, and the model storage unit 51 and the action determination mechanism. Notify the unit 52.

Further, the sensor input processing section 50 has an image recognition section 50B, and the image recognition section 50B performs image recognition processing using the image signal given from the video camera 16. Then, when the image recognition unit 50B detects, for example, "a red round object" or "a plane perpendicular to the ground and having a predetermined height or more" as a result of the processing,
Image recognition results such as "there is a red ball" and "there is a wall" are notified to the model storage unit 51 and the action determination mechanism unit 52 as state recognition information.

Further, the sensor input processing section 50 has a pressure processing section 50C, and the pressure processing section 50C processes the pressure detection signal given from the touch sensor 17. Then, as a result of the processing, the pressure processing unit 50C recognizes that the pressure is equal to or higher than a predetermined threshold and for a short period of time, and "recognizes that the pressure has been hit," and is less than the predetermined threshold, Further, when the pressure for a long time is detected, it is recognized as "stroked (praised)", and the recognition result is notified to the model storage unit 51 and the action determination mechanism unit 52 as state recognition information.

The model storage unit 51 stores and manages an emotion model, an instinct model, and a growth model that express the emotion, instinct, and growth state of the robot.

The emotion model is, for example, "joy",
The emotional states (degrees) such as “sadness”, “anger”, and “joy” are set in a predetermined range (for example, −1.0 to 1.0).
Etc.) and the sensor input processing unit 50
The value is changed on the basis of the state recognition information from, the elapsed time, and the like. The instinct model is, for example, the state (degree) of the desire by the instinct such as “appetite”, “sleep desire”, “exercise desire”
Are represented by values in a predetermined range, and the values are changed based on the state recognition information from the sensor input processing unit 50, the elapsed time, and the like. The growth model represents, for example, growth states (degrees) such as “childhood”, “adolescence”, “mature”, “old age” by a value in a predetermined range, and outputs from the sensor input processing unit 50. The value is changed based on the state recognition information of, the time passage, and the like.

The model storage unit 51 sends the emotion, instinct, and growth state represented by the values of the emotion model, the instinct model, and the growth model as state information to the action determination mechanism unit 52.

The model storage section 51 is supplied with the state recognition information from the sensor input processing section 50, and the action determination mechanism section 52 provides the current or past behavior of the robot, specifically, for example, " Behavior information indicating the content of the behavior such as “walked for a long time” is supplied, and the model storage unit 51 recognizes the behavior of the robot 1 indicated by the behavior information even if the same state recognition information is given. Different state information is generated accordingly.

That is, for example, when the robot 1 greets the user and pats the head on the user, the action information indicating that the user greets the user and the state recognition information indicating that the head is patted on the user are displayed. The value is given to the model storage unit 51, and in this case, the value of the emotion model representing “joy” is increased in the model storage unit 51.

On the other hand, when the robot 1 pats its head while performing some work, the model storage unit stores the action information indicating that the job is being carried out and the state recognition information indicating that the head is patted. 51, and in this case, in the model storage unit 51, the value of the emotion model representing “joy” is not changed.

As described above, the model storage unit 51 sets the value of the emotion model with reference to not only the state recognition information but also the action information indicating the current or past action of the robot 1. This causes an unnatural emotional change, such as increasing the value of the emotional model expressing "joyfulness" when the user pats his / her head with the intention of mischief while performing some task. You can avoid that.

The model storage unit 51 is adapted to increase or decrease the values of the instinct model and the growth model based on both the state recognition information and the behavior information, as in the case of the emotion model. Further, the model storage unit 51 is configured to increase or decrease the values of the emotion model, the instinct model, and the growth model based on the values of other models.

The action determination mechanism unit 52 determines the next action based on the state recognition information from the sensor input processing unit 50, the state information from the model storage unit 51, the passage of time, etc. The content is sent to the posture transition mechanism unit 53 as action command information.

That is, the action determination mechanism unit 52 manages a finite state automaton in which actions that the robot can take correspond to states, as an action model that defines the action of the robot. State in the finite state automaton as a sensor input processing unit 5
Based on the state recognition information from 0, the value of the emotion model, the instinct model, or the growth model in the model storage unit 51, the elapsed time, and the like, the action corresponding to the state after the transition is determined as the action to be taken next. To do.

The action determination mechanism section 52 uses a predetermined trigger (tri
gger) is detected, the state is transited. That is, the action determination mechanism unit 52 is supplied from the model storage unit 51, for example, when the time during which the action corresponding to the current state is executed reaches a predetermined time or when the specific state recognition information is received. When the value of the emotion, instinct, or growth state indicated by the state information becomes equal to or less than or equal to a predetermined threshold value, the state is transited.

As described above, the action determination mechanism section 52 is based on not only the state recognition information from the sensor input processing section 50 but also the values of the emotion model, the instinct model, the growth model, etc. in the model storage section 51. Since the states in the behavior model are transited, even if the same state recognition information is input, the transition destinations of the states are different depending on the values (state information) of the emotion model, the instinct model, and the growth model.

As a result, for example, when the state information indicates "not angry" and "not hungry", the action determination mechanism section 52 determines that the state recognition information indicates "eyes". When it indicates that the palm was held out in front, the action command information that causes the action “hand” to be taken is generated in response to the palm held out in front of the eye, and this is It is sent to the posture transition mechanism unit 53.

When the state recognition information indicates that the voice recognition result of "follow the red ball" is obtained, action command information for causing the action of "chasing the red ball" is generated, which is the posture. It is sent to the transition mechanism unit 53.

Further, the action determining mechanism unit 52, for example,
When the state information indicates that "I am not angry" and "I am hungry", and when the state recognition information indicates that "the palm of my hand is in front of me" , And generates action command information for performing an action of “licking the palm” in response to the palm being held in front of the eye, and sends this to the posture transition mechanism unit 53.

In addition, for example, when the state information indicates "angry", the action determining mechanism section 52 indicates that the state recognition information is "the palm was put out in front of the eyes". When it is displayed, even if the status information indicates that you are "hungry" or "not hungry", you should behave like "Looking sideways". The action command information to be performed is generated and sent to the posture transition mechanism unit 53.

As described above, the action determination mechanism section 52 also generates action command information for causing the robot 1 to speak, in addition to the action command information for operating the head, limbs, etc. of the robot 1. The action command information that causes the robot 1 to speak is supplied to the voice synthesizer 55.
The action command information supplied to the voice synthesizing unit 55 includes a text or the like corresponding to the synthetic sound generated by the voice synthesizing unit 55. Then, the voice synthesizing unit 55 includes the action determining mechanism unit 5
When the action command information is received from 2, the synthetic sound is generated based on the text included in the action command information and is supplied to the speaker 18 to be output. As a result, the speaker 18
From, for example, a crying voice of the robot 1, various requests to the user such as "hungry", a response to the user's call such as "what?", And other voice outputs are performed. In addition, the action determination mechanism unit 52 outputs action command information for opening and closing the lower jaw portion 4A when outputting a synthetic sound,
It is generated as needed and is output to the posture transition mechanism unit 53.
In this case, the lower jaw 4A is opened and closed in synchronization with the output of the synthetic sound, and the user can be given an impression as if the robot 1 is speaking.

The posture transition mechanism section 53 includes the action determination mechanism section 5
Based on the action command information supplied from the robot 1, the robot 1
The posture change information for changing the posture of the current posture from the current posture to the next posture is generated and sent to the control mechanism unit 54.

The control mechanism section 54 operates the actuator 3AA in accordance with the posture transition information from the posture transition mechanism section 53.
₁ to 3DA _K, generates a control signal for driving the 5A ₁ and 5A _2, this actuator 3AA ₁ to 3
Send to DA _K , 5A ₁ and 5A ₂ . As a result, the actuators 3AA _{1 to} 3DA _K , 5A ₁ and 5A
₂ drives according to a control signal, and the robot 1 autonomously takes action.

FIG. 4 shows a configuration example of the voice recognition unit 50A and the image recognition unit 50B of FIG.

The audio signal from the microphone 15 is AD (Analo
g Digital) is supplied to the conversion unit 21. AD converter 21
Performs sampling and quantization of a voice signal which is an analog signal from the microphone 15 and AD-converts it into voice data which is a digital signal. This voice data is the feature extraction unit 22.
Is supplied to.

The feature extraction unit 22 extracts, for example, the MF of the audio data input thereto for each appropriate frame.
CC (Mel Frequency Cepstrum Coefficient) analysis is performed, and the MFCC obtained as a result of the analysis is output to the matching unit 23 and the unregistered word section processing unit 27 as a feature vector (feature parameter). In addition, the feature extraction unit 22 can also extract, for example, a linear prediction coefficient, a cepstrum coefficient, a line spectrum pair, a power (output of a filter bank) for each predetermined frequency band, as a feature vector.

The matching unit 23 uses the feature vector from the feature extraction unit 22 to input to the microphone 15 while referring to the acoustic model storage unit 24, the dictionary storage unit 25, and the grammar storage unit 26 as necessary. Voice (input voice)
Is speech-recognized based on, for example, the continuous distribution HMM (Hidden Markov Model) method.

That is, the acoustic model storage unit 24 represents an acoustic model representing acoustic characteristics of individual phonemes in the language of the speech to be recognized, subwords such as syllables and phonemes (for example, HMM, DP (Dynamic Programming)). ), Including standard patterns used for matching). Note that, here, since the voice recognition is performed based on the continuous distribution HMM method, an HMM (Hidden Markov Model) is used as the acoustic model.

In the dictionary storage unit 25, information on the pronunciation of the word (phonological information), which is clustered for each word to be recognized, and the headline of the word are associated with each other, and if necessary, Further, a word dictionary in which information (RGB values) related to images is associated is stored.

FIG. 5 shows an example of the word dictionary 71 stored in the dictionary storage unit 25.

As shown in FIG. 5, in the word dictionary 71, word headings are associated with their phoneme sequences, and the phoneme sequences are clustered for each corresponding word. In the word dictionary 71 of FIG. 5, one entry (one line in FIG. 3) corresponds to one cluster.

In the word dictionary 71 of FIG. 5, ak
For the heading a [red], the RGB value is (215, 0
35,093) is supported, and the heading of orenji [orange] is (167,095,056).
RGB values are supported.

In FIG. 5, headings are shown in Roman letters and Japanese (Kana / Kanji), and phoneme sequences are shown in Roman letters. However, "N" in the phoneme sequence
Represents the sound repellency "n". Further, in FIG. 5, one phoneme sequence is described in one entry, but a plurality of phoneme sequences can be described in one entry.

Returning to FIG. 4, the grammar storage unit 26 stores grammar rules describing how the words registered in the word dictionary of the dictionary storage unit 25 are linked (connected).

FIG. 6 shows an example of the grammar rules 81 stored in the grammar storage unit 26. Note that the grammar rule 81 in FIG.
Is described in EBNF (Extended Backus Naur Form).

In FIG. 6, from the beginning of the line to the first appearance of ";" represents one grammar rule. In addition, alphabets (columns) prefixed with "$" represent variables, and alphabets (columns) without "$"
Indicates a word heading (heading in Roman letters shown in FIG. 5). Further, the portion enclosed by [] indicates that it can be omitted, and “|” indicates that one of the words (or variables) of the headings arranged before and after it is selected.

Therefore, in FIG. 6, for example, the grammar rule “$ col = [kono | sono] ir” on the first line (first line from the top) is used.
"o wa;" indicates that the variable $ col is a word string of "this color (color) is" or "that color (color)".

In the grammar rule 81 shown in FIG. 6, the variables $ sil and $ garbage are not defined, but the variable $ s
il represents a silent acoustic model (silent model), and the variable $ g
Arbage basically represents a garbage model that allows free transitions between phonemes.

Returning to FIG. 4 again, the matching unit 23 refers to the word dictionary 71 of the dictionary storage unit 25 to connect the acoustic models stored in the acoustic model storage unit 24, thereby connecting the acoustic models of the words. (Word model). Further, the matching unit 23 connects some word models by referring to the grammatical rules 81 stored in the grammar storage unit 26, and uses the word models thus connected based on the feature vector. , Continuous distribution HMM
By the method, the voice input to the microphone 15 is recognized.
That is, the matching unit 23 detects the sequence of the word model having the highest score (likelihood) at which the time-series feature vector output from the feature extraction unit 22 is detected, and the word string corresponding to the sequence of the word model is detected. The headline is output as a voice recognition result.

More specifically, the matching unit 23 accumulates the appearance probabilities (output probabilities) of the respective feature vectors with respect to the word strings corresponding to the connected word models, and sets the cumulative value as a score. The headline of the word string to be made the highest is output as the voice recognition result.

Further, the matching section 23 includes the image recognition section 5
The above-described matching process is executed by using the feature vector of the image extracted by the 0B feature extraction unit 42 together with the feature vector of the voice extracted by the feature extraction unit 22 as necessary.

The microphone 15 output as described above
The recognition result of the voice input to the
It is output to the model storage unit 51 and the action determination mechanism unit 52.

In the embodiment shown in FIG. 6, the ninth line (from the top 9
Line), a grammar rule using the variable $ garbage that represents the garbage model (hereinafter, referred to as an unregistered word rule) "$ pat
1 = $ color1 $ garbage $ color2; ", but the matching unit 23 determines that when this unregistered word rule is applied,
The voice section corresponding to the variable $ garbage is detected as the voice section of an unregistered word. Further, the matching unit 23 detects a phoneme sequence as a phoneme transition in the garbage model represented by the variable $ garbage when the unregistered word rule is applied, as the phoneme sequence of the unregistered word. Then, the matching unit 23 supplies the unregistered word segment processing unit 27 with the unregistered word speech segment and the phoneme sequence that are detected when the speech recognition result to which the unregistered word rule is applied is obtained.

The above unregistered word rule "$ pat1 = $ c
According to "olor1 $ garbage $ color2;", the phoneme sequence of the word (column) registered in the word dictionary represented by the variable $ color1 and the phoneme sequence represented by the variable $ color2 registered in the word dictionary. Although one unregistered word between the word (string) and the phoneme sequence is detected, in the present invention, when the utterance includes a plurality of unregistered words or when the unregistered word is a word dictionary. It is applicable even when it is not sandwiched between words (columns) registered in.

The unregistered word section processing unit 27 includes the feature extraction unit 2
The sequence of feature vectors of the voice (feature vector sequence) supplied from No. 2 is temporarily stored. Further, when the unregistered word section processing unit 27 receives the unregistered word speech section and the phoneme sequence from the matching unit 23, the unregistered word section processing unit 27 detects the speech feature vector series in the speech section from the temporarily stored feature vector series. . Then, the unregistered word section processing unit 27 attaches a unique ID (Identification) to the phoneme sequence (unregistered word) from the matching unit 23 to identify the phoneme sequence of the unregistered word and the feature vector sequence in the speech section. It is supplied to the feature vector buffer 28 together with (voice feature vector series).

The feature vector buffer 28, for example, as shown in FIG. 7, temporarily stores the IDs of unregistered words supplied from the unregistered word section processing unit 27, phoneme sequences, and speech feature vector sequences in association with each other. To do.

In FIG. 7, a sequential number from 1 is added to the unregistered word as an ID.
Therefore, for example, when the feature vector buffer 28 is now storing N unregistered word IDs, phoneme sequences, and phonetic feature vector sequences, the matching unit 23 causes the unregistered word voice segments and phoneme sequences to be stored. When an unregistered word is detected, the unregistered word section processing unit 27 assigns N + 1 to the unregistered word as an ID, and the feature vector buffer 28 indicates the unregistered word of the unregistered word as indicated by a dotted line in FIG. The ID, phoneme sequence, and voice feature vector sequence are stored.

Returning to FIG. 4 again, the image recognition unit 50B displays the A
It has a D converter 41. The AD converter 41 samples and quantizes an image signal which is an analog signal input from the video camera 16 and A / D converts it into image data which is a digital signal. This image data is supplied to the feature extraction unit 42.

The feature extraction unit 42 calculates, for example, an average value of RGB values in a predetermined range of the image data input thereto, and the obtained result is used as a feature vector (feature parameter) of the image to perform matching. It is supplied to the unit 23 and the feature vector buffer 43. The predetermined range to be sampled is a predetermined range in the center of the screen, a range designated by the user, or the like. Alternatively, the feature extraction unit 42 receives the information indicating the voice section of the unregistered word from the matching unit 23, and detects the feature vector of the image corresponding to the voice section.

The feature vector extracted by the feature extraction unit 42 is supplied to the feature vector buffer 43 and stored therein. The feature vector of the image stored in the feature vector buffer 43 is further supplied to the feature vector buffer 28 and stored as an image feature vector series. As shown in FIG. 7, this image feature vector sequence is stored in correspondence with the phoneme sequence and the voice feature vector sequence.

The clustering unit 29 uses the unregistered words newly stored in the feature vector buffer 28 (hereinafter, as appropriate,
Feature vector buffer 2 for new unregistered words)
A score for each of the other unregistered words already stored in 8 (hereinafter, appropriately referred to as an already stored unregistered word) is calculated.

That is, the clustering unit 29 regards the new unregistered word as the input voice and the already stored unregistered word as the word registered in the word dictionary, and in the same manner as in the matching unit 23, For the new unregistered word, a score for each stored unregistered word is calculated. Specifically, the clustering unit 29 recognizes the feature vector series of the new unregistered word by referring to the feature vector buffer 28, and connects the acoustic models in accordance with the phoneme series of the stored unregistered word. From the connected acoustic model, a score is calculated as the likelihood that the feature vector sequence of the new unregistered word is observed.

The acoustic model is stored in the acoustic model storage unit 2.
The one stored in 4 is used.

In the same way, the clustering unit 29 also calculates a score for the new unregistered word for each already stored unregistered word, and the score sheet storage unit 30 is calculated according to the score.
Update the score sheet stored in.

Further, the clustering unit 29 uses both the voice feature vector sequence and the image feature vector sequence when calculating the score.

Specifically, the clustering unit 29 sets, for example, a score calculated based on the voice feature vector series to Sa, a score calculated based on the image feature vector series to Sv, and sets α and β to predetermined values, respectively. When the coefficient is, the score Sc synthesized according to the following equation is calculated, and clustering is performed using this synthesized score Sc.

[0094] Sc = α × Sa + β × Sv (1)

The coefficients α and β are coefficients for weighting the score by the voice and the score by the image, and are appropriately changed as necessary.

The score Sv can be calculated by, for example, RGB
The value of Euclidean distance can be used.

The clustering unit 29 also refers to the updated score sheet to create a new unregistered word from among the clusters of unregistered words (already stored unregistered words) that have already been obtained. Discover the clusters to add as members. Furthermore, the clustering unit 2
9 designates the new unregistered word as a new member of the detected cluster, divides the cluster based on the members of the cluster, and based on the division result, the score stored in the score sheet storage unit 30. Update the sheet.

The score sheet storage unit 30 stores a score sheet in which a score for a new unregistered word, a score for a previously stored unregistered word, a score for a previously unregistered word for a new unregistered word, and the like are registered. .

FIG. 8 shows an example of the score sheet.

The score sheet 91 includes the unregistered word "ID",
"Phonological series", "Cluster number", "Representative member I"
It is composed of entries in which "D" and "score" are described.

As the unregistered words “ID” and “phoneme sequence”, the same ones stored in the feature vector buffer 28 are registered by the clustering unit 29.
The “cluster number” is a number for identifying a cluster in which the unregistered word of the entry is a member, is assigned by the clustering unit 29, and is registered in the score sheet 91. The "representative member ID" is an ID of an unregistered word as a representative member representing a cluster in which the unregistered word of the entry is a member, and the unregistered word is a member by this representative member ID. The representative member of the cluster can be recognized. The representative member of the cluster is obtained by the clustering unit 29, and the ID of the representative member is registered in the representative member ID of the score sheet 91. The “score” is the score of the unregistered word of the entry for each of the other unregistered words, and is calculated by the clustering unit 29 as described above.

For example, assuming that the feature vector buffer 28 stores N unregistered word IDs, phoneme sequences, and feature vector sequences as shown in FIG. As shown in FIG. 8, the N unregistered word IDs, phoneme sequences, cluster numbers, representative member IDs, and scores are registered.

Then, as shown by the dotted line in FIG.
When a new unregistered word ID, a phoneme sequence, and a feature vector sequence are newly stored in the feature vector buffer 28, the clustering unit 29 updates the score sheet 91 as indicated by the dotted line in FIG. 8.

That is, in the score sheet 91, the score of the new unregistered word ID, the phoneme sequence, the cluster number, the representative member ID, and the new unregistered word for each of the stored unregistered words (score s ( N + 1,1), s (N + 1,2), ..., s
(N + 1, N)) is added. Further, in the score sheet 91, the score for the new unregistered word for each of the stored unregistered words (s (1, N + 1), s (2, N + 1), ...
,, s (N, N + 1)) is added. Further, as will be described later, the cluster numbers and representative member IDs of unregistered words on the score sheet 91 are changed as necessary.

In the embodiment of FIG. 8, the ID is
For the unregistered word (utterance of i) of i, the score for the unregistered word (phoneme sequence of) of ID j is represented as s (i, j).

The score sheet 91 (FIG. 8) has an ID
The score s (i, i) for the unregistered word of (i.e., the utterance of) is also registered for the unregistered word of (i.e., its phoneme sequence). However, this score s (i, i) is
Since the calculation is performed when the phoneme sequence of the unregistered word is detected, the clustering unit 29 does not need to calculate.

Returning to FIG. 4 again, the maintenance section 31
The word dictionary 71 stored in the dictionary storage unit 25 is updated based on the updated score sheet 91 in the score sheet storage unit 30.

The representative member of the cluster is determined as follows. That is, for example, among the unregistered words that are members of the cluster, the total sum of the scores of the other unregistered words (otherwise, for example, an average value obtained by dividing the total sum by the number of other unregistered words may be used. ) Is the representative member of the cluster. Therefore, in this case, if the member ID of the member belonging to the cluster is represented by k, the member having the value K (εk) represented by the following equation as the ID is the representative member.

K = max _k {Σs (k ', k)} (2)

However, in the equation (2), max _k {} means k that maximizes the value in {}. Also, k'means the ID of a member belonging to the cluster, similar to k. further,
Σ means the sum of k ′ varied over the IDs of all members belonging to the cluster.

When the representative member is determined as described above, when the members of the cluster are 1 or 2 unregistered words, it is not necessary to calculate the score when determining the representative member. That is, if the members of the cluster are 1
In the case of two unregistered words, the one unregistered word becomes the representative member, and when the cluster member is two unregistered words, which of the two unregistered words is the representative member? Good as a member.

The method of deciding the representative member is not limited to the above-described one, but other than that, for example, among the unregistered words that are members of the cluster, the feature vector of each of the other unregistered words. The one that minimizes the total sum of distances in space may be the representative member of the cluster.

The voice recognition section 50A configured as described above.
Then, the voice recognition process for recognizing the voice input to the microphone 15 and the unregistered word process regarding the unregistered word are performed using the image captured by the video camera 16.

Therefore, first, the voice recognition processing will be described with reference to the flowchart of FIG.

When the user speaks, the spoken voice is input in step S1. That is, microphone 1
5 and the digital voice data input through the AD conversion unit 21 are supplied to the feature extraction unit 22. Further, in step S2, the image signal obtained as a result of being imaged by the video camera 16 is input, and AD
AD conversion is performed in the conversion unit 41, and the converted data is supplied to the feature extraction unit 42.

In step S3, the feature extraction unit 22 extracts a feature vector by acoustically analyzing the voice data in a predetermined frame unit, and the feature vector series is processed by the matching unit 23 and the unregistered word section processing. It is supplied to the unit 27. The feature extraction unit 42 also performs step S4.
In, the feature vector is extracted by analyzing the image data, and the series of the feature vector is supplied to the matching unit 23.

In step S5, the matching unit 23 described above is based on the feature vector sequence (voice feature vector sequence) from the feature extraction unit 22 and the feature vector sequence (image feature vector sequence) from the feature extraction unit 42. As described above, the score is calculated using the equation (1), and the process proceeds to step S6. In step S6, the matching unit 23
Based on the score obtained as a result of the score calculation, the headline of the word string that becomes the voice recognition result is obtained and output.

Further, the matching section 23 carries out step S
In step 7, it is determined whether the user's voice contains an unregistered word.

In step S7, the voice of the user is
When it is determined that the unregistered word is not included, that is, the rule for unregistered word “$ pat1 = $ color1 $ garbage $ colo
r2; ”is not applied and the speech recognition result is obtained,
Step S8 is skipped and the process ends.

If it is determined in step S7 that the user's voice includes an unregistered word, that is, the unregistered word rule "$ pat1 = $ color1 $ garbage $ colo".
r2; ”is applied to obtain a speech recognition result, the process proceeds to step S8, and the matching unit 23 determines that the speech section corresponding to the variable $ garbage of the unregistered word rule is the unregistered word speech section. Along with the detection, the phoneme sequence as the transition of the phoneme in the garbage model represented by the variable $ garbage is detected as the phoneme sequence of the unregistered word, and the phoneme section and the phoneme sequence of the unregistered word are unregistered word section processing unit. 27, and the process ends.

On the other hand, the unregistered word section processing unit 27 temporarily stores the feature vector series supplied from the feature extraction section 22, and when the matching section 23 supplies the unregistered word speech section and the phoneme series. , The feature vector sequence of the voice in the voice section is detected. Further, the unregistered word section processing unit 27 assigns an ID to (the phoneme sequence of) the unregistered word from the matching unit 23, and, together with the phoneme sequence of the unregistered word and the feature vector sequence in the speech section, a feature vector buffer 28.

Further, the feature extraction unit 42 is the matching unit 2
The feature of the image data corresponding to the voice section of the unregistered word detected by 3 is extracted, and the feature vector thereof is supplied to the matching unit 23 and also supplied to the feature vector buffer 43 to be stored. The feature vector is also supplied from the feature vector buffer 43 to the feature vector buffer 28 and stored therein.

As described above, when the feature vector buffer 28 stores the ID of a new unregistered word (new unregistered word), the phoneme sequence, and the feature vector sequence of voice and image, the unregistered word process is performed. Is done.

FIG. 10 is a flow chart for explaining the unregistered word process in this case.

In the unregistered word processing, first, in step S11, the clustering unit 29 reads the ID and phonological sequence of the new unregistered word from the feature vector buffer 28, and proceeds to step S12.

In step S12, the clustering unit 2
9 refers to the score sheet 91 of the score sheet storage unit 30 to determine whether there is a cluster that has already been obtained (generated).

If it is determined in step S12 that the already-obtained cluster does not exist, that is, the new unregistered word is the first unregistered word, and the score sheet 9
If the entry of the stored unregistered word does not exist in 1, the clustering unit 29 creates a cluster having the new unregistered word as a representative member, in step S13.
Information regarding the new cluster and information regarding the new unregistered word are stored in the score sheet 9 of the score sheet storage unit 30.
The score sheet 91 is updated by registering 1 in the score sheet 91.

That is, the clustering unit 29 registers the ID and phonological sequence of the new unregistered word read from the feature vector buffer 28 in the score sheet 91 (FIG. 8). Further, the clustering unit 29 generates a unique cluster number and registers it in the score sheet 91 as a cluster number of a new unregistered word. Also, the clustering unit 2
9 is the ID of the new unregistered word and the representative member of the new unregistered word
The ID is registered in the score sheet 91. Therefore, in this case, the new unregistered word becomes the representative member of the new cluster.

In this case, since there is no stored unregistered word for calculating the score with the new unregistered word, the score is not calculated.

After the processing of step S13, step S2
2, the maintenance unit 31 determines the dictionary storage unit 25 based on the score sheet 91 updated in step S13.
The word dictionary 71 of is updated, and the process ends.

That is, in this case, since a new cluster has been generated, the maintenance unit 31 refers to the cluster number in the score sheet 91 and recognizes the newly generated cluster. And the maintenance section 3
1 adds the entry corresponding to the cluster to the word dictionary 71 of the dictionary storage unit 25, and as the phoneme sequence of the entry, the phoneme sequence of the representative member of the new cluster, that is, new unregistered in this case. Register the phoneme sequence of a word.

On the other hand, when it is determined in step S12 that the already-obtained cluster exists, that is, the new unregistered word is not the first unregistered word, and therefore the score sheet 91 (FIG. 8) shows If there is an entry (row) of the stored unregistered word, the process proceeds to step S14,
The clustering unit 29 calculates a score for each already-stored unregistered word for the new unregistered word by using the above-described formula (1), and at the same time, for each already-stored unregistered word, scores for the new unregistered word To calculate.

That is, for example, if there are N stored unregistered words with IDs 1 to N and the ID of the new unregistered word is N + 1, the clustering unit 29 uses the dotted line in FIG. Scores s (N + 1,1), s (N + 1,2), ..., s for each of the N already-stored unregistered words for the new unregistered word in the indicated portion
(N + 1, N) and the scores s (1, N + 1), s (2, N + 1), ... for the new unregistered words for each of the N stored unregistered words
s (N, N + 1) is calculated. Note that in the clustering unit 29, in order to calculate these scores, the feature vector series of each of the new unregistered word and the N stored unregistered words is required. It is acquired by referring to 28.

Then, the clustering unit 29 calculates the calculated score together with the ID of the new unregistered word and the phoneme sequence.
The score sheet 91 is added to the score sheet 91 (FIG. 8), and the process proceeds to step S15.

In step S15, the clustering unit 2
9 refers to the score sheet 91 (FIG. 8) to obtain the score s (N + 1, i) (i = 1,2, ...
-, N) find the cluster with the representative member that makes (N) the highest (large). That is, the clustering unit 29 refers to the representative member ID of the score sheet 91,
By recognizing the stored unregistered words that are the representative members and further referring to the score of the score sheet 91, the stored unregistered words as the representative members that maximize the score of the new unregistered words are detected. To do. Then, the clustering unit 29 detects the cluster of the cluster number of the stored unregistered word as the detected representative member.

Then, in step S16, the clustering unit 29 adds the new unregistered word to the members of the cluster detected in step S15 (hereinafter, appropriately referred to as a detected cluster). That is, the clustering unit 29 writes the cluster number of the representative member of the detected cluster as the cluster number of the new unregistered word on the score sheet 91.

Then, in step S17, the clustering unit 29 performs cluster division processing for dividing the detected cluster into, for example, two clusters (details of the processing will be described later with reference to FIG. 11), and in step S18.
Proceed to. In step S18, the clustering unit 29
Determines whether or not the detected cluster can be divided into two clusters by the cluster division processing in step S17, and if it is determined that the detected cluster can be divided, the process proceeds to step S19. In step S19, the clustering unit 29 determines two clusters obtained by dividing the detected clusters (hereinafter, these two clusters will be referred to as the first cluster as appropriate).
Inter-cluster distance between two child clusters (referred to as a child cluster and a second child cluster).

The inter-cluster distance between the first and second child clusters is defined as follows, for example.

That is, the IDs of arbitrary members (unregistered words) of both the first child cluster and the second child cluster are represented by k, and the representative members of the first and second child clusters (unregistered words) are represented. If the IDs of the words are represented by k1 or k2, respectively, the value D (k1, k2) represented by the following equation is the inter-cluster distance between the first and second child clusters.

D (k1, k2) = maxval _k {abs (log (s (k, k1))-log (s (k, k2)))} (3)

However, in the equation (3), abs () represents the absolute value of the value in (). Also, maxval _k {} represents the maximum value in {} obtained by changing k. Also, log represents natural logarithm or common logarithm.

Now, if the member with ID i is represented as member #i, the reciprocal of the score in equation (3) 1 / s (k, k
1) corresponds to the distance between the member #k and the representative member k1, and the reciprocal 1 / s (k, k2) of the score corresponds to the distance between the member #k and the representative member k2. Therefore, according to equation (3), the first and second
Of the members of the child cluster of the first child cluster and the representative member # k1 of the second child cluster, and the representative member of the second child cluster #
The maximum value of the difference with the distance from k2 is the inter-child cluster distance between the first and second child clusters.

As described above, the score and the distance have an inverse relationship, but since they correspond one-to-one, it is possible to use the distance as the score. However, in this case, the magnitude relationship is reversed.

The inter-cluster distance is not limited to the one described above, but other than that, for example, DP matching between the representative member of the first child cluster and the representative member of the second child cluster is performed. With, the integrated value of the distance in the feature vector space is obtained, and the integrated value of the distance is
The distance between clusters can also be used.

After the processing of step S19, step S2
Proceeding to 0, the clustering unit 29 determines whether the inter-cluster distance between the first and second child clusters is larger than a predetermined threshold value θ (or is equal to or larger than the threshold value θ).

In step S20, when it is determined that the inter-cluster distance is larger than the predetermined threshold value θ, that is, a plurality of unregistered words as members of the detected cluster are
If it is considered that the two clusters should be clustered because of their acoustic characteristics, step S
21, the clustering unit 29 sets the first and second child clusters to the score sheet 9 of the score sheet storage unit 30.
Register to 1.

That is, the clustering unit 29 assigns unique cluster numbers to the first and second child clusters, and among the members of the detected cluster, the cluster number of the one clustered to the first child cluster is set to the first cluster number. And the cluster number of the one clustered to the second child cluster to the cluster number of the second child cluster.
The score sheet 91 is updated.

Further, the clustering unit 29 is a representative member of the members clustered into the first child cluster.
Score sheet so that the ID is the ID of the representative member of the first child cluster and the representative member ID of the member clustered in the second child cluster is the ID of the representative member of the second child cluster Update 91.

The cluster number of the detected cluster can be assigned to either one of the first and second child clusters.

When the clustering unit 29 registers the first and second child clusters in the score sheet 91 as described above, the process proceeds from step S21 to S22, and the maintenance unit 31 calculates the score sheet 91 based on the score sheet 91. The word dictionary 71 in the dictionary storage unit 25 is updated, and the process ends.

That is, in this case, the detected cluster is the first
Since it is divided into the second child cluster, the maintenance unit 31 first deletes the entry corresponding to the detected cluster in the word dictionary 71. In addition, the maintenance unit 31 has 2 nodes corresponding to each of the first and second child clusters.
One entry is added to the word dictionary 71, the phoneme sequence of the representative member of the first child cluster is registered as the phoneme sequence of the entry corresponding to the first child cluster, and the entry corresponding to the second child cluster. The phoneme sequence of the representative member of the second child cluster is registered as the phoneme sequence of the entry.

On the other hand, in step S18, the detected clusters are divided into two by the cluster division processing in step S17.
When it is determined that the cluster cannot be divided into two clusters, or when it is determined in step S20 that the inter-cluster distance between the first and second child clusters is not larger than the predetermined threshold θ (hence, If the acoustic features of the plurality of unregistered words as members of the detected cluster are not so similar as to cluster them into the first and second child clusters), the process proceeds to step S23, where the clustering unit 29 A new representative member of the detected cluster is obtained and the score sheet 91 is updated.

That is, the clustering unit 29 refers to the score sheet 91 of the score sheet storage unit 30 for each member of the detected cluster to which the new unregistered word is added as a member, and is required for the calculation of the equation (2). Score s (k ',
Recognize k). Further, the clustering unit 29 uses the recognized score s (k ′, k) to obtain the ID of the member to be the new representative member of the detected cluster based on the equation (2). Then, the clustering unit 29 uses the score sheet 9
The representative member ID of each member of the detection cluster in 1 (FIG. 8) is rewritten to the ID of a new representative member of the detection cluster.

After that, the procedure goes to step S22, in which the maintenance section 31 updates the word dictionary 71 in the dictionary storage section 25 based on the score sheet 91, and the processing ends.

That is, in this case, the maintenance unit 31
Recognizes a new representative member of the detected cluster by referring to the score sheet 91, and further recognizes the phoneme sequence of the representative member. And the maintenance section 3
1 changes the phoneme sequence of the entry corresponding to the detected cluster in the word dictionary 71 to the phoneme sequence of the new representative member of the detected cluster.

Details of the cluster division processing in step S17 of FIG. 10 will be described below with reference to the flowchart of FIG.

In the cluster division process, first, in step S31, the clustering unit 29 selects an arbitrary combination of two members that have not been selected from the detected clusters to which the new unregistered word has been added as a member. , And each is a temporary representative member. Here, these two temporary representative members will be appropriately referred to as a first temporary representative member and a second temporary representative member hereinafter.

Then, in step S32, the clustering unit 29 sets the two members of the detected cluster to two representative members so that the first temporary representative member and the second temporary representative member can be respectively set as the representative members. Determine if it can be divided into clusters.

Here, whether or not the first or second temporary representative member can be the representative member needs to be calculated by the equation (2), but the score used for this calculation
s (k ′, k) is acquired by referring to the score sheet 91.

In step S32, the number of detected cluster members is set to 2 so that the first temporary representative member and the second temporary representative member can be set as the representative members.
When it is determined that the cluster cannot be divided into one cluster, step S33 is skipped and the process proceeds to step S34.

In step S32, the member of the detected cluster may be divided into two clusters so that the first temporary representative member and the second temporary representative member can be set as the representative members. If it is determined that it is possible, the process proceeds to step S33, and the clustering unit 29
Divides the member of the detected cluster into two clusters such that the first temporary representative member and the second temporary representative member are representative members, respectively, and sets the set of the two clusters after the division. The process proceeds to step S34 as first and second child cluster candidates (hereinafter, appropriately referred to as a set of candidate clusters) that are the result of the division of the detected cluster.

In step S34, the clustering unit 2
9 is still the first and second among the members of the detected cluster.
It is determined whether or not there is a set of two members that have not been selected as the set of temporary representative members of No., and if it is determined that there is a set of members, the process returns to step S31 and still selected as the set of first and second temporary representative members. A pair of two members of the detected cluster that have not been selected is selected, and the same processing is repeated thereafter.

If it is determined in step S34 that there is no set of two members of the detected cluster that has not been selected as the set of the first and second temporary representative members, the process proceeds to step S35 and the clustering unit 29 Determines whether a set of candidate clusters exists.

If it is determined in step S35 that there is no candidate cluster set, step S36 is skipped and the process returns. In this case, in step S18 of FIG. 10, it is determined that the detected cluster could not be divided.

On the other hand, if it is determined in step S35 that there is a set of candidate clusters, step S36.
Proceeding to step, the clustering unit 29 obtains the inter-cluster distance between two clusters of each candidate cluster set when there are a plurality of candidate cluster sets. And
The clustering unit 29 obtains a set of candidate clusters having the smallest inter-cluster distance, and returns the set of candidate clusters as the division result of the detected clusters, that is, as the first and second child clusters. The set of candidate clusters is 1
If there are only two, the set of candidate clusters is
They are the first and second child clusters.

In this case, it is determined in step S18 of FIG. 10 that the detected cluster can be divided.

As described above, the clustering unit 29 detects the cluster (detection cluster) to which the new unregistered word is added as a new member from the clusters obtained by clustering the unregistered words, which have already been obtained. The unregistered word is divided into new members of the detected cluster, and the detected cluster is divided based on the members of the detected cluster. , Can be easily clustered.

Further, since the maintenance unit 31 updates the word dictionary based on such a clustering result, unregistered words can be registered in the word dictionary while avoiding an increase in the size of the word dictionary. It can be done easily.

Further, for example, even if the matching section 23 makes a mistake in detecting the voice section of an unregistered word,
Such unregistered words are
The voice segment is clustered into a cluster different from the unregistered word for which the correct detection is performed. Then, an entry corresponding to such a cluster will be registered in the word dictionary, but since the phoneme sequence of this entry corresponds to the speech section that was not correctly detected, in the subsequent speech recognition. , Don't give a big score. Therefore, even if the detection of the voice section of the unregistered word is erroneous, the error hardly affects the subsequent voice recognition.

Further, when not only the feature vector of the voice but also the feature vector of the image is synthesized and the clustering is performed using the synthesized feature vector, compared to the case of using only the feature vector of the voice. Therefore, clustering can be performed more accurately. As a result, it becomes possible to perform voice recognition more accurately.

Next, with reference to the flow chart of FIG. 12, the action processing performed in the robot 1 learned as described above based on the feature vector based on the voice and the image will be described.

Now, for example, it is assumed that the user speaks to the robot 1 "follow the red ball".

In step S51, the microphone 15 inputs this voice signal from the user. This audio signal is
In step S52, voice recognition is performed by the voice recognition unit 50A.

In step S53, the action determining mechanism section 52 determines whether or not the image data (RGB value) is registered, corresponding to the result of the voice recognition in step S52. In this case, as shown in the word dictionary 71 of FIG. 5, since the RGB value is stored corresponding to the word “red”, it is determined that the image data is registered.

In this case, the process proceeds to step S55, and the image signal picked up by the video camera 16 is input. The image recognition unit 50B performs image recognition processing of the image data input in step S55, and outputs the recognition result to the action determination mechanism unit 52. The action determining mechanism unit 52, step S
At 56, a process of specifying a voice input target from the input image data based on the registered image data is executed. Specifically, in this case, since the RGB value is registered corresponding to the word “red”, the action determination mechanism unit 52 registers the RGB value from the recognition result output from the image recognition unit 50B. RGB value equal to or close to the existing RGB value
A range of images having a value is recognized as a “red” image. In this case, since the "red ball" is the target object, the action determination mechanism unit 52 extracts a circular image from the "red" images and recognizes this as the "red ball". .

In step S57, the action determination mechanism section 52 executes the process corresponding to the voice input for the target specified in the process of step S56. In the present case,
The voice recognition unit 5 recognizes "following" the "red ball".
Since it is performed by 0A, the action determination mechanism unit 52
Posture transition mechanism 53 so as to chase the "red ball"
To control. The posture transition mechanism unit 53 further controls the control mechanism unit 54 based on this control, and the actuator 3
AA ₁ to 3DA _K, of 5A ₁ and 5A _2, drives a predetermined actuator. As a result, the robot 1
You will be chasing the "red ball".

In this case, the recognition of "red" is performed by the image recognition unit 50.
Since the image recognition by B is also used, it becomes possible to follow the “red ball” more accurately than in the case of recognition only by voice.

If it is determined in step S53 that the image data is not registered corresponding to the recognition result, the process proceeds to step S54, and the process corresponding to the voice input is executed. That is, in this case, since the recognition result based on the image is not used, the same action based on the recognition result as in the conventional case is executed.

The robot 1 cannot be made to recognize the "red ball" by recognizing only the voice.

Also, for example, "red" and "orange"
If the word and the image data thereof are registered in advance, the “red ball” can be identified as the “orange ball” by processing the image data input from the video camera 16. “Red” and “orange” are relatively similar colors, and if recognition processing is performed only from image data, the robot 1 may mistakenly recognize an “orange ball” as a “red ball”. .

However, as described above, when the recognition processing is performed by using both the voice and the image, even if the colors (RGB values) of "red" and "orange" are similar, "red" Since the pronunciation "and" is not similar to the pronunciation "orange", the robot 1 can accurately identify the "red ball" and the "orange ball".

Even in the case of clustering the words "red" and "orange", it is possible to accurately cluster them by using not only images but also sounds. It is also a result of this clustering that the robot 1 can accurately identify the “red ball” and the “orange ball”.

In the above, the case where the colors are learned based on the sound and the image and clustered and registered is described as an example. However, for example, when the words “increase” and “decrease” are clustered and registered. The present invention can be applied.

That is, the phoneme of "raise" is "ageru".
And the phoneme of "lower" becomes "sageru". The latter has only a fricative "s" at the beginning, and the other phonemes are the same as the former.

The fricative "s" is a phoneme that is relatively difficult to identify due to background noise. For this reason,
If the words "increase" and "decrease" are clustered only by the voice and registered in the dictionary, there is a possibility that they may be clustered in the same cluster even though they are completely different words.

However, not only the voice but, for example,
If the operation of "raising" or "lowering" a predetermined object is captured by the video camera 16 at the same time as the voice and registered corresponding to the voice, the operations of "raising" and "lowering" are completely opposite operations. Therefore, be sure to distinguish between
It is possible to cluster into different clusters. In this case, a motion vector can be used as the feature vector of the "increase" and "decrease" images. The motion vector of the "raising" image is a vector heading upward, whereas the motion vector associated with the "lowering" image is a vector heading downward. Therefore, it is possible to easily cluster the both into exactly different clusters.

Furthermore, the present invention can also be applied to the case of clustering and registering a person's name.

For example, when the voice "This is <OOV>" is input, the phoneme of the <OOV> portion corresponding to this unregistered word can be clustered and registered. For example, when the voice "This is Mr. Yamada" is input, if "Mr. Yamada" is not registered in the dictionary, "Mr. Yamada" will be registered in the dictionary. At this time, for example, an image of “Ms. Yamada” from the front can be simultaneously captured and registered in the dictionary in correspondence with the phoneme of “Ms. Yamada”.

[0189] Alternatively, also, "Hello. My name is Yamada." Such as "Hello. My name is <OO
V> When the voice of the pattern is input, <O
The words in the portion of OV> may be clustered and registered together with the images that have been input at the same time.

In this way, for example, when the phonemes and images of "Ms. Yamada" are clustered and registered, when the "Ms. Yamada" is subsequently introduced to another person, the robot 1 The scene can be picked up by the video camera 16 and, for example, an image of “Mr. Yamada” seen from the side can be sampled and registered. Even if it is the same "Mr. Yamada", the image from the front and the image from the side are completely different images. Therefore, for example, the robot 1 is caused to perform the recognition process only from the image,
Further, when the clustering and registration processing is executed, it is difficult for the robot 1 to recognize the image in the front and the image from the side as the image of the same person. Therefore, the clustering and the registration are performed as different persons. On the other hand, as in the case of the present invention, if the image is registered in association with the voice, the robot 1 will not be registered as "Mr. Yamada" unless another "Mr. Yamada" with the same surname is registered. From the phoneme, it is possible to recognize that the two images (front image and side image) are related (to be exact, the same) person's image.

In the above description, the image data is registered in the dictionary. However, a dedicated storage unit may be provided and registered in the dictionary.

The case where the present invention is applied to a robot for entertainment (robot as a pseudo pet) has been described above. However, the present invention is not limited to this, and for example, a voice dialogue system equipped with a voice recognition device, etc. It can be widely applied to. Further, the present invention is applicable not only to a robot in the real world, but also to a virtual robot displayed on a display device such as a liquid crystal display.

Further, in the present embodiment, the series of processes described above is executed by causing the CPU 10A to execute a program, but the series of processes can also be executed by dedicated hardware. .

Here, the program is stored in the memory 10B (FIG. 2) in advance, as well as a floppy (registered trademark) disk or a CD-ROM (Compact Disc Read Only Memo).
ry), MO (Magneto optical) disc, DVD (Digital Vers
It can be temporarily or permanently stored (recorded) in a removable recording medium such as an atile disc), a magnetic disc, or a semiconductor memory. Then, such a removable recording medium can be provided as so-called package software and installed in the robot (memory 10B).

Further, the program may be transferred wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or may be transferred by wire via a network such as a LAN (Local Area Network) or the Internet to store the program in the memory 10B. Can be installed on.

In this case, when the program is upgraded, the upgraded program can be easily installed in the memory 10B.

In the present specification, the processing steps in which the programs for causing the CPU 10A to perform various processes are described do not necessarily have to be processed in time series in the order described in the flow charts. It also includes individually executed processing (for example, parallel processing or object processing).

The program may be processed by one CPU or may be processed by a plurality of CPUs in a distributed manner.

Further, the voice recognition unit 50A shown in FIG. 4 can be realized by dedicated hardware or software. Voice recognition unit 50A
When the software is realized by software, a program forming the software is installed in a general-purpose computer or the like.

Therefore, FIG. 13 shows an example of the configuration of an embodiment of a computer in which a program for realizing the voice recognition unit 50A is installed.

The program is stored in the hard disk 105 or the ROM 1 as a recording medium built in the computer.
03 can be recorded in advance.

Alternatively, the program is temporarily or permanently stored (recorded) in a removable recording medium 111 such as a floppy (registered trademark) disk, a CD-ROM, an MO disk, a DVD, a magnetic disk, or a semiconductor memory. Can be set. Such a removable recording medium 11
1 can be provided as so-called package software.

The program is installed in the computer from the removable recording medium 111 as described above, and is also transferred wirelessly from the download site to the computer via an artificial satellite for digital satellite broadcasting, LAN, Internet, etc. The program can be transferred to a computer via a network via a network, and the program can be received by the communication unit 108 and installed in the built-in hard disk 105 in the computer.

The computer is a CPU (Central Processing).
Unit) 102 is built in. CPU 102 has a bus 1
The input / output interface 110 is connected via 01, and the CPU 102 operates the input unit 107 including a keyboard, a mouse, a microphone, an AD converter, etc. by the user via the input / output interface 110. When a command is input as a result, the program stored in the ROM (Read Only Memory) 103 is executed accordingly. Alternatively, the CPU 102 also
The program stored in the hard disk 105, the program transferred from the satellite or the network, received by the communication unit 108, and installed in the hard disk 105, or read from the removable recording medium 111 mounted in the drive 109 and stored in the hard disk 105. Installed programs are stored in RAM (Random
It is loaded into the Access Memory) 104 and executed.

As a result, the CPU 102 performs the processing according to the above-described flowchart or the processing performed by the configuration of the above-described block diagram. And CPU1
Reference numeral 02 designates the processing result of the LCD (Liquid Crystal) via the input / output interface 110, if necessary.
Display, etc., speakers, DA (Digita)
l Analog) Output from the output unit 106 configured by a converter or the like, or transmitted from the communication unit 108, and further recorded on the hard disk 105.

In the present embodiment, the voice recognition is performed by the HMM method, but the present invention is also applicable to the voice recognition by the DP matching method or the like. Here, for example, when performing voice recognition by the DP matching method, the above-mentioned score corresponds to the reciprocal of the distance between the input voice and the standard pattern.

In the present embodiment, the unregistered words are clustered and the unregistered words are registered in the word dictionary based on the clustering result. However, the present invention does not register the unregistered words in the word dictionary. It is also applicable to registered words.

That is, even if the same word is uttered, different phoneme sequences may be obtained. Therefore, when only one phoneme sequence is registered for one word in the word dictionary, When a phoneme sequence different from the phoneme sequence of the registered word registered in the word dictionary is obtained as the utterance of the word, the utterance may not be recognized by the registered word. On the other hand, according to the present invention, different utterances for the same word are clustered acoustically similar to each other. Therefore, by updating the word dictionary based on the clustering result. , For the same word, various phoneme sequences will be registered in the word dictionary, and as a result, for the same word,
It is possible to perform voice recognition that deals with various phonemes.

[0209] In addition to the phonological sequence, for example, a heading can be described as follows in the entry corresponding to the cluster of unregistered words registered in the word dictionary.

That is, for example, in the action determination mechanism section 52, the state recognition information output by the image recognition section 50B and the pressure processing section 50C is supplied to the voice recognition section 50A as indicated by the dotted line in FIG. The maintenance unit 31 (FIG. 4) of the voice recognition unit 50A receives the state recognition information.

On the other hand, the feature vector buffer 28, and by extension, the score sheet storage unit 30, also stores the absolute time (time) at which the unregistered word is input, and the maintenance unit 31 stores the score sheet storage unit 30. By referring to the absolute time of the score sheet in
The state recognition information supplied from the action determination mechanism unit 52 when the unregistered word is input is recognized as the heading of the unregistered word.

Then, in the maintenance section 31, the entry corresponding to the cluster of unregistered words in the word dictionary is registered with the phoneme sequence of the representative member of the cluster and the state recognition information as its heading.

In this case, the matching unit 23 can output the state recognition information as the headline of the unregistered word as the voice recognition result of the unregistered word registered in the word dictionary. It is possible to make the robot take a predetermined action based on the state recognition information.

In the present embodiment, the score is stored in the score sheet, but the score may be recalculated if necessary.

Further, in the present embodiment, the detection cluster is divided into two clusters, but the detection cluster can be divided into three or more clusters. Further, the detection cluster can be divided into an arbitrary number of clusters having a certain inter-cluster distance or more.

Further, in the present embodiment, in addition to the score, the phoneme sequence of the unregistered word, the cluster number, the representative member ID, etc. are registered in the score sheet (FIG. 8).
Information other than these scores can be managed separately from the scores instead of being registered in the score sheet.

[0217]

According to the present invention, input information can be clustered. Further, according to the present invention, input information can be clustered more easily.
Further, according to the present invention, input information can be clustered more accurately. Therefore, according to the present invention, the input information can be registered and recognized more accurately.

Further, according to the present invention, it is possible to avoid a large scale due to registration of registration information.

[Brief description of drawings]

FIG. 1 is a perspective view showing an external configuration example of an embodiment of a robot to which the present invention is applied.

FIG. 2 is a block diagram showing an internal configuration example of the robot of FIG.

FIG. 3 is a block diagram showing a functional configuration example of the controller of FIG.

4 is a block diagram showing a configuration example of a voice recognition unit in FIG.

FIG. 5 is a diagram showing an example of a word dictionary.

FIG. 6 is a diagram showing an example of grammar rules.

FIG. 7 is a diagram showing an example of stored contents of a feature vector buffer of FIG.

FIG. 8 is a diagram showing an example of a score sheet.

FIG. 9 is a flowchart illustrating a voice recognition process.

FIG. 10 is a flowchart illustrating unregistered word processing.

FIG. 11 is a flowchart illustrating a cluster division process.

FIG. 12 is a flowchart illustrating action processing.

FIG. 13 is a block diagram showing a configuration example of an embodiment of a computer to which the present invention has been applied.

[Explanation of symbols]

1 head unit, 4A lower jaw, 10 controller, 10A CPU, 10B memory, 15
Microphone, 16 video camera, 17 touch sensor, 18 speaker, 21 AD converter, 22
Feature extraction unit, 23 matching unit, 24 acoustic model storage unit, 25 dictionary storage unit, 26 grammar storage unit,
27 unregistered word section processing unit, 28 feature vector buffer, 29 clustering unit, 30 score sheet storage unit, 31 maintenance unit, 42 feature extraction unit, 43 feature vector buffer, 50 sensor input processing unit, 50A speech recognition unit, 50B image Recognition unit, 50C pressure processing unit, 51 model storage unit,
52 behavior determination mechanism unit, 53 posture transition mechanism unit, 54
Control mechanism part, 55 speech synthesis part, 101 bus,
102 CPU, 103 ROM, 104 RAM, 10
5 hard disk, 106 output section, 107 input section, 108 communication section, 109 drive, 11
0 input / output interface, 111 removable recording medium

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G06T 7/00 300 G10L 3/00 521C G10L 13/00 551H 15/00 571Q 15/12 533Z 15/14 535Z 15/20 Q 15/24 531Q F term (reference) 3C007 AS36 CS08 KS31 KS39 KT01 KX02 WA04 WA14 WB16 WB19 5D015 CC01 CC07 CC11 HH07 HH23 KK01 5D045 AB11 5L096 BA16 FA00 FA32 JA11 KA03 MA07

Claims (8)

[Claims] 1. An information processing apparatus for recognizing input information based on registration information, comprising: first acquisition means for acquiring first information; and second acquisition means for acquiring second information. A first feature extracting means for extracting a feature of the first information obtained by the first obtaining means; and a second feature extracting means for extracting a feature of the second information obtained by the second obtaining means. The feature of the first information extracted by the first feature extracting unit, and the feature of the second information extracted by the second feature extracting unit, Clustering means for clustering the first information acquired by the first acquisition means, and registration means for registering the first information as the registration information based on the result of clustering by the clustering means. An information processing device characterized by the above. 2. The clustering means includes the feature of the first information extracted by the first feature extracting means and the second feature extracted by the second feature extracting means.
The information processing apparatus according to claim 1, wherein the first information is clustered on the basis of the characteristics obtained by weighting and adding the characteristics of the information. 3. The first information and the second information are one of voice and an image, and the other.
The information processing device according to 1. 4. The first or second feature extraction means,
The information processing apparatus according to claim 3, wherein the characteristics of the image are extracted as RGB values. 5. The information processing apparatus according to claim 3, wherein the first information is a voice of an unregistered word that is not registered as the registration information. 6. An information processing method of an information processing apparatus for recognizing input information based on registration information, comprising a first acquisition step of acquiring first information and a second acquisition step of acquiring second information. An acquisition step, a first feature extraction step of extracting a feature of the first information acquired by the process of the first acquisition step, and a second feature acquired by the process of the second acquisition step. A second feature extracting step of extracting a feature of information; a feature of the first information extracted by the process of the first feature extracting step; and a feature extracted by the process of the second feature extracting step. Using the second information feature,
The first information is registered as the registration information based on a clustering step of clustering the first information acquired by the processing of the first acquisition step and a result of clustering by the processing of the clustering step. An information processing method comprising: a registration step. 7. A program of an information processing device for recognizing input information based on registration information, the first acquisition step of acquiring first information, and the second acquisition step of acquiring second information. A first feature extraction step of extracting a feature of the first information acquired by the processing of the first acquisition step; and a second feature extraction step of the second information acquired by the processing of the second acquisition step. A second feature extraction step of extracting features, a feature of the first information extracted by the process of the first feature extraction step, and a second feature extracted by the process of the second feature extraction step Using the information features of
The first information is registered as the registration information based on a clustering step of clustering the first information acquired by the processing of the first acquisition step and a result of clustering by the processing of the clustering step. A recording medium having a computer-readable program recorded thereon, the recording medium including a registration step. 8. A program executable by a computer that controls an information processing apparatus that recognizes input information based on registration information, the program including a first acquisition step of acquiring first information, and a second information. A second acquisition step of acquiring, a first characteristic extraction step of extracting a characteristic of the first information acquired by the processing of the first acquisition step, and a second characteristic acquisition step of the second acquisition step A second characteristic extracting step of extracting a characteristic of the second information, a characteristic of the first information extracted by the processing of the first characteristic extracting step, and a processing of the second characteristic extracting step Using the characteristics of the second information extracted by
The first information is registered as the registration information based on a clustering step of clustering the first information acquired by the processing of the first acquisition step and a result of clustering by the processing of the clustering step. A program comprising a registration step.