TW200411627A - Robottic vision-audition system - Google Patents

Abstract

A robottic vision-audition system includes an audition module (20), a face module (30), a stereo module (37), a motor control module (40), and an association module (50) for controlling each module. The audition module (20) includes an active direction passing type filter (23a) having a pass range which varies, according to correct information of sound source direction from the association module (50), such that the pass range is minimum at a front direction and is enlarged with the enlargement of the angles in right and left directions, to perform a sound source separation by collecting subbands having an Inter-ears Phase Difference (IPD) or an Inter-ears Intensity Difference (IID) and re-constructing the wave forms of the rounds from source. The audition module (20) further performs a recognition of sounds separated from the sounds from the source, by recognising sound signals separated by a plurality of audeo models (27d),consolidating the results of sound recognitions of each audeo module by selector (27e), and determining the most reliable sound recognition result from these sound recognition results.

Description

200411627 _, description of the invention [Technical field to which the invention belongs] The present invention relates to an audio-visual system suitable for robots, especially human-type or animal-type robots. [Prior technology] In recent years, the above-mentioned human or animal robots are not only the object of AI (Anificial Intelligence) research purposes, but they are also regarded as so-called "human partners" in the future. In order for robots to perform intelligent social activities (soccial interactions), perceptions such as hearing and hearing are necessary for robots. Moreover, it is clear that in order to realize social activities with humans, the visual and auditory consciousness, especially the auditory consciousness, is an important function. Therefore, more and more attention has been paid to vision and hearing, so-called active perception. Here, the so-called active perception refers to the purpose of following the goal of a sensory device that can sense the perception of the robot such as robot vision or robot hearing, for example, by driving the mechanism to support the aforementioned sensory head to follow the target to control Poser. In the robot's active vision, = the camera of the perceptual device (cam is called the lack of ^ by the drive mechanism, keeps its silk direction toward the target, and further-step by step automatically ^^., (F〇CUsing) ^, ^ (z〇〇m (zoom out), etc.) This way, even if the target moves, you can still take pictures. In the past, quoting photography scenes has introduced various of the above active visions. On the other hand, the machine is crying at the Xi Shifang Station < research. You Shi ^ Yu Jieren's active hearing, at least knowing the strong and strong gt; Keep its J M4] 9 6 200411627 directivity, and use the microphone to collect the sound from the target with active hearing, and the ear θ to collect sound. For the disadvantages of this, the driving mechanism When the microphone is used, it will receive the driving broadcast ... m During the action of the structure, 'due to the dynamic sound of the movement', therefore, the corresponding sound will be mixed with 4 * 丄, @ before showing the large noise, resulting in unrecognizable targets. In order to overcome the shortcomings of active hearing, m Here is how to trace back the sound of hunting the sounds of McCaw. The original ... way to correctly identify the target is not == in the above-mentioned active hearing, according to the sound collected by the microphone, you can use the sound collected by the microphone. From each sound source, each sounding separation is f PF, plus L two power seven Ah. I 虿 (c) identify the sound from each sound source. It is (A) sound source localization and (b) sound Tracking points '; — 9, Yuan Dali has entered into various researches on active source hearing in true time and real environment — 3 sources of position, tracking, and separation under y. For example,' such as International Publication No. 01 / 953 14 Revealed by the Tiger Bulletin, the interaural phase difference (interaural phase ruggedness) and interaural Intensity Differenee nD obtained from HRTF (Head Related Transfer Functións) were used. ) To perform sound source localization research is well known. Moreover, in the above-mentioned literature, it is generally known to use, for example, a so-called directional pass-type filter, ie, an Erection-Pass Filter, by selecting an ipD with a specific direction. Same IP The D subband is a method for separating the sounds from the various sound sources. In contrast, regarding the identification of the sounds from the sound sources separated by the sound source, for example, for-) or missing data ) And other noise methods for robust identification (Robust) 3.144J2 200411627 sound recognition methods, various studies have been carried out as shown in the following documents. J. Baker et al. 'Robustness based on clear speech model (c 1 ean speech m〇de 1)', "European Speeches 2001-The 7th European Conference Proceedings," 2001, "Brother 1 Volume" pp, 213 -216 (J. Baker, M. Cooke, and P. Green, Robust as based on clean speech models: An evaluation of missing data techniques for connected digit recognition in noise · 7th European conference on Speech Communication Lu Technology ", Volume 1, p. 213-216). P. Renebe et al. Robust Speech Recognition, "European Lectures 200, 7th European Conference Proceedings", 2001 "Brother 12" pp.ll〇7-llll〇 (philippe Renevey, Rolf

Vetter, and Jens Kraus. Robust speech recognition using missing featuie theory and vector quantization. U7th

Euiopean Confeience on Speech Communication Technology ,,, Volume 12, pp. 1107-1110). Ref. However, in the studies disclosed in the above two documents, when the S / N ratio is small, speech recognition cannot be performed effectively. In addition, research on speech recognition in a real-life, real-world environment has not yet been conducted. [Summary of the Invention] The present invention has been developed in view of the above problems, and an object thereof is to provide a robotic audiovisual system capable of recognizing separated sounds from various sound sources. In order to achieve the above a, the first component of the robotic audiovisual system of the present invention includes: a combination of the vocabulary sent by each speaker and its direction, and a plurality of audeo models (314412 8 200411627); using the above Acoustic model,-Perform a speech recognition process w rec〇gniti〇n Pr〇cesO on the sound signal that has been separated from the sound source; and, through the speech recognition process, a plurality of speeches obtained by the sound model A selector that integrates the results of the recognition process to select any one of the results of the voice recognition process; it is characterized by the ability to identify the sons and daughters of each speaker simultaneously. The cosmetic selector is constructed by selecting the results of the aforementioned speech recognition process by a majority decision method, and may also include a dialogue in which the results of the speech recognition process selected by the comment selector are rotated to the outside. unit. The method of using multiple sound models is to use the sound signals of the first source's foot position and sound separation, and do not perform the voice recognition process. Then, through the selector, the results of the speech recognition process obtained from the various voice shapes are integrated to determine the reliable speech recognition results. In addition, 'to achieve the above-mentioned objective, the robot audiovisual system composition system of the present invention' is provided with: at least 2: which collects external sounds; ;; ': According to the acoustic signal emitted by the microphone, Degree (pnch) extraction, harmonic structure separation of the eighth sound source, and determination of n_n (㈣ coffee g), and the direction of the Gong speaker, and the auditory module of its I and auditory events; With a camera 嫱 ^: a camera shirt machine in front of a person, according to the camera's recognition and positioning, identify each speaker: two 'face module by howling of each speaker; with horizontal The facial event (fa direction to drive the robot 3144) 2 9 200411627 The motor 'draws out the motor control module according to the rotational position of the driving motor; from the aforementioned auditory matter. The complete piece (aud event 0n event), face contention, and motor event, determine the direction of each a Ό ϋ brother according to the direction information of the auditory event η / the original position and the age location of the facial event, Using Kalman filter (Kalman filter), the aforementioned events are connected in a time direction to generate an auditory data stream and a facial data stream, and then, related to the above data stream to generate a combination module (association stream) ( ass〇Clation module); and Zhuyin control based on these data streams, and 'attention control module for motor drive control based on the actions generated by the accompanying attention control (p. 2 results, the attention control module n consciousness module is based on the The correct sound source 2 from the aforementioned combination module :; the signal, which is the most positive in the front direction and changes as the left and right angles increase: the active direction pass filter of the pass range of the pass band will have a predetermined width The ear-to-ear phase difference (IpD) or second ear difference (IID) within the amplitude range is collected, and the waveform of the sound source is reconstructed to separate the sound sources. At the same time, a plurality of Loud-The voice recognition of the sound signals separated by the sound source is performed by the selector. The selector recognizes the result of the voice recognition according to each sound plan, and judges among the 7 distinguished sounds of these words. Consisting of the most reliable speech recognition result. According to the above-mentioned second configuration, the hearing module is the sound received by the microphone and from the outside, and the harmonic extraction is used to extract the spacing, such as ®. The direction of each sound source to identify each speaker, ^ ^ ^ field and speech. Moreover, the face module is identified by the pattern of the day and two hearts captured by the camera. The face recognition and positioning of each speaker, ° Mu extracts each of the words 314412 10 200411627 speaker's face event. Furthermore, the motor control module is based on the rotation position of the drive motor that causes the robot to rotate in the horizontal direction. The direction of the robot is detected in order to extract the motor event. Among them, the above-mentioned event refers to the detection of deafness, sound, or the face of the face at each time point, θ _ 1 ^ ^ or 疋Do not drive the motor in a state of rotation; the so-called data clean _, Tt * (st! Earn) are not only the error correction process, but the Kdman filter (Kdman filter) and other time-continuous methods to continue Here, the combination module is based on the auditory events, facial events, and motor events that are extracted by the above methods to generate the auditory training material stream and facial data stream of each speaker, and then, Correlation results in the combination of lean feed streams, and the attention control module. (Attention claw claws. Such as e) Attention control according to the above data flow (attentiQn e. Alas. 1) In order to control the driving motor of the motor control module, the day is calculated. In the middle, the ^ combined data stream system includes the right _ camp: Bei Ke… see the concepts of Bei Yi stream and facial data stream. In addition, the idea of H means that the robot will “listen” to the target to hear and / or visually. The so-called attention-control motive control module changes its direction. Note that the control module controls the drive motor of the electric group according to the day and time, and will "" Wu Xiangzhi speakers. In this way, the robot is facing the works :: Xiangxiang == pair =: the speaker's voice can be sensitive by using the microphone ::: the correct collection and localization of the direction of the speaker; and the camera's day image of the speaker Make good photography. Hunting photo 314412 2⑻411627 Therefore, with the cooperation of the above-mentioned hearing module, face module module, combination module, and attention protection &quot; Eight You-Rotary System ~ Ligong Zhizhu Group, you can make your ears shake. And visual complementation have their own chanting M so that the eight people of the ancestors are not born, and the two so-called robustness ft ') can be felt by each speaker, even by multiple speakers. And 'Even if, for example, any one of the auditory event or the facial event is missed,' as long as the facial events and sensory events are combined, the module can generate knowledge of the target speaker. Real-time control of the motor control module for entering the lamp. ^ Furthermore, the aforementioned auditory module has already performed sound source localization according to the above. ::! Knife of the sound f tiger ’uses a plurality of sound models to perform speech recognition respectively. Pick up! — Stomach ^ Wrong. The selector recognizes the speech recognition results paid for each acoustic model. The first, the first &amp; In this way, * Bai Zhizhi-sound recognition is compared, by using a plurality of acoustic models, γ, # — Λ τ in the true time and real environment for correct speech recognition, the same as _, inch 'By combining the recognition result of each acoustic model by the selector, μ Douda u judges the result of the most reliable speech recognition, and can perform more accurate speech recognition. In addition, in order to achieve the above object, the 3rd component of the robotic audiovisual system of the present invention is provided with: at least a pair of microphones that collect sounds from external sounds, and send out according to the microphones. By determining the direction of at least one ^ ^ person who is speaking, and by extracting the sound source based on the south of the sound and the harmonic structure, ^ v .. grouping, the sound source is _ and positioning, and the | -〇'See the auditory module of the event; ^ ^ ^ The camera in front of Jie Ren is taking pictures according to the daytime image taken by the camera. The face of each speaker is 314412 12 200411627 II. '4 different § brother talkers' and extract his face The facial model of the event is based on the parallax extracted from the day image captured by the stereo camera, and extracts and locates the longer longitudinal object, and draws the carving # 4 module of the stereo event): The rotation position of the driving motor is extracted from the motor control module of the motor =; from the aforementioned auditory event, facial event, standing == motivation event, according to the sound source localization of the auditory event and the direction information of the facial event board position For the direction of each speaker, a kalman filter is used to connect the aforementioned events in the α time direction, &amp; _ :, 疋 generates ^, sensational material stream, facial data stream, and stereo vision data. ， Combined with the above-mentioned data stream to produce a combination of combined data stream and the attention control based on the above-mentioned data stream, and the motor drive control based on the action plan results generated by the system. Group; where the aforementioned auditory module is based on the correct sound source direction information of the combined module, which is the smallest in the front direction, and the passing frequency range (pass that increases with the left and right angles) Lange) 's active direction pass filter adds the sub-bands with inter-ear phase difference (IPD) or inter-ear intensity difference (IID) within a predetermined wide range, and reconstructs the waveform of the sound source. In order to perform sound source separation, at the same time, when performing speech recognition, a plurality of sound models are used for speech recognition of the sound signals separated from the sound sources, and the speech recognition results obtained according to each sound model are integrated by a selector. Among the results of speech recognition, the result of the speech recognition with the highest reliability is determined. According to the third configuration above, the auditory module is collected by the microphone 314412] 3 200411627 The sound from the external target is used to obtain each -I wave structure to extract the sound height, the direction of the vowel source, and the female And extract the directions of I_ and t and tQ speakers. In the day image, the P ^, 且, and heart, ^ identify ^ 7% A position of each speaker obtained from the line pattern captured by the camera, Distinguish 夂% t 4. The fc part can identify and identify the individual speakers, and extract each speaker's three-dimensional handhold # h event. Furthermore, ',,,,' are based on the parallax extracted by the stereo camera from the longitudinal image with the parallax extracted from the image, the long object is pulled out and positioned, and the grapefruit motor makes the H button. Li Yuedou incident. Furthermore, the rotation position of the moving motor is a driving event of the forward rotation of the forward lamp. garrison. . The direction of the person, and the motor is drawn out. The so-called event is the sound detected by the judge, and it indicates that there is movement at each point of time = the machine is rotating. , "Yue η ... Ding Zheng dealt with, while, for example, card and cry filter) and other events such as the Japanese fighting from Cong Shi and 'Balkoukou (Kalman, Yue Shengsheng) succession in succession. Here, the combined model The group is based on the sound source location of the auditory parts extracted from the <Nai-style knife> and the facial location of the facial events based on the above events, hopes, and burials using the direction of the speaker of the auditory event to the funeral! Let's decide each of them to produce the auditory data stream, face Qiu-bei material stream and three-dimensional data stream of each speaker. However, the personal data, related to gangan, beke / melon are related, and: ^. Among them, the so-called combined data stream includes the concept of "aural stream and stereoscopic data stream." At this time, the combination of two: sound source localization according to auditory events and face localization of facial events, two ::; Decide on each speech based on the direction of hearing and vision The direction of the person = 314412 14 200411627, the direction of each of the five brothers to generate the combined capital n [control system 'attention control module is based on the above data flow H, move ::, and according to the accompanying attention control generated Result: Motivation-driven control. Then, 'Note that the control module is oriented toward the speaker as the target according to c. # Put the "people's square target speaker, auditory handwork" state person is facing the work as a group of fans. Use the microphone to

In the positive direction with high sensitivity, Zheng-Li stands “the voice of the brother 5 tongues Qiu Lu 4 smashes into the sound collection and positioning, and at the same time, the Japanese / V # group can use photography As a result, due to the above listening ~ "backed up good photography. Motor control module, combination module: and branch group, three-dimensional module, and stand-by D ::, ”according to auditory data, and attention to the cooperation of control module, the root direction is: two: ... position and facial data flow The speaker positioning and other visual complementation are self-righteous, and ΐΓ mistakenly makes the robot's hearing and multiple ambiguities, and improves the so-called robustness, even if only the knife does not actually produce perception of each speech. Moreover, even if for example omission Any one of the audio data stream can be said to be W Beco ~ and the three-dimensional meeting can be based on the residual data stream, and it can be followed by the note, and the tracking is ... Ϋ—Congratulations, so you can correctly grasp the control of the target party's motive control module. Here, the 'audience module' refers to the facial information from the face module at the same time as the social person; :::: The combination of data flow and material flow for sound source localization uses the stereo vision of two == and:, and the aforementioned auditory mode is based on the sound source position with L. The source direction information is based on the auditory characteristics and combines the correctness of the module. The sound is minimal in the front direction, As the angle of the left 314412 15 becomes larger, the passband range of the main and forward pass filters will have both ears within a predetermined wide range = (IPD) or the intensity difference between the two ears. ⑽) The sub-bands are collected, and the waveforms of the heavy, ten, and articulated sources are separated for sound source separation. Therefore, ϋ adjust the pass frequency _ range, which is the sensitivity method, according to the upper limit L, and: Due to the difference in sensitivity caused by the side and direction, a more accurate separation of the source is performed. Ran, · ^, +, y, _ 1 place I sense module has been determined by the auditory sound source as described above彳 立 ,, // 5 八 μa ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, by, by using the plurality of sound batches ^ 仃 ^ The ancient and ugly edgings obtained by Crying in accordance with each acoustic model can be integrated. The mouth and mouth are combined to determine the corresponding speech recognition result, and the speech recognition result is compared with Fortunately, it is related. In this way, the sound recognition with the known voice is really made by using a plurality of sounds. The correct speech recognition of the down lights, and at the same time, the various speech recognition and results are combined by the selector to determine the most reliable heart ~ fruit, and more accurate speech recognition can be performed. Among them, in the second The composition and Chu 1 de Λ, &gt; +, the group performs attention to distinguishing,… during the composition '▲ cannot be identified by the auditory model, a note control module uses the acoustic signal ::: standing towards the aforementioned microphone and the aforementioned camera, From the aforementioned wheat again: the wind sounds the sound of the sound, Xi Xiu and the sound of the sound, separation, and separation of the sound module made by the auditory module-, Le Jing &lt; The sound was again voiced by the auditory module. Yindan ^ See the microphone and face camera of the facial module are facing the speaker _ If there is a mistake, it can be accurately identified. "When the auditory module performs speech recognition, it is advisable to refer to the facial events obtained by the facial module J6 3344J2 200411627. Moreover, ... The judged speech recognition result is turned to =: It is equipped with the aforementioned auditory module All moving directions pass through the filter to the active angle. Furthermore, the aforementioned main control is suitable. The over-frequency range is such that the aforementioned auditory module can be performed at each frequency. When performing human identification & sound recognition, μ a ▲ The way of combining the tributary stream, the combination of the fine and the Mai Zhao module, that is, the general, the Sizhi Department of the face Shenmu, ancient. The group n is about Also, it is based on the acoustic signals of the facial events of A 'and' 仃 疋 position by the auditory module to perform speech recognition 1. The recognition of the separated sound source (speaker). When the aforementioned active abbot can perform more When the correct speech recognition frequency is controlled, the passband range of the 2 / state can be further improved according to the sounds collected by each J. This can also improve the speech recognition. "I. departure accuracy" by [Implementation] The following according to the figure The first and second figures shown in the implementation are shown ...: : The present invention's system - only form of embodiment the upper body "human Cheng Li audiovisual system. In the first figure of the first figure of the humanoid robot, the humanoid robot 10 # hibiscus 4 degrees, flutters gree of f ^, and is known as a 4 DOF (freedom of freedom) robot, with a seat. On the base 1 ~ 1, the base 1 1; the left and right directions of the moon, which are supported at the base ... and the rotation direction, and on the carcass portion 12 on the carcass portion 12 in a three-axis direction, and the horizontal axis is white. Water-backed head of the first 10 cars) way of shaking 7 Shuo 13. The above-mentioned base U can be fixedly configured for moving bamboos or can be provided with feet. Moreover, the base can also be mounted on the base. As shown in Figure ^, it is placed on a movable trolley "^" shown in the arrow "A" in the figure, "the body part is attached to the base u 314412 17 in a rotatable manner around the vertical axis k, and the rotary target is turned over沾 η ## Driven by a driving mechanism not shown in the figure. At the same time, the icon is shown to cover it again. The outer packaging is anti-sung, and the 3rd part is connected to the body part 12 through the first mu. , And ... The component Ha is supported, as shown by the front number B in the figure, and the connecting component Wach # ^ in the direction of Aλ 3 a may be, 'nine of them are shaken in front of the thousand axis, and as shown in FIG. 2 Lu + Jiayi said that as shown by C, the horizontal axis of the left / right direction is shaken as shown by the arrow D in the figure 1. The above door body 12 can be set to the front part of the ancient German 々 连 连 member 13a.胴 ，, Μ 々 々 + + means for the shaking method. An unillustrated driving mechanism, ^, the main Dinggu bluffs A, B, C, and price to drive forward. Here, the head 12 is driven by Entering the carving θ, as shown in Figure 3, the burial is made by the king ’s body with sound-proof packaging on the wrong side as A, and it is self-contained, and it has: in front., ΒΪ ^ 见 祝 祝 见 衣Camera, 1.5, and the right side as the responsible machine, crying at lfin, and pee σ. One of the hearing devices of human hearing is the microphone! 6 (16 &amp; 16b). Langzhong,, Xi Xiong paper side, The star festival of Asia and Africa 16 is limited to the two of the head 13, and it can also be set on the head, the, the other position, the carcass part 12, etc. The above-mentioned outsourcing I 1 4 is made by the amino acid Licong—Human version of TS 夂 B® urethane resin and other synthetic resins, which are almost completely opened and closed inside the Shuo Bu 13 for Fujie ··. Head 4 13 It is composed of internal sound-shielding methods. Among them, the outer package of the carcass body 12 is also made of sound-absorbing synthetic resin in the same way as the head σP 1 3 Do not _ 1/1 turn 14. The above camera 15 passes rabbits Idle is a well-known structure, which is a commercially available camera with, for example, the so-called horizontal pan, pan, Tilt, and Zoom 3 DOFf Hakusanjiro, Yudo). The camera 3144J2] 8 200411627 l5 series is designed to transmit stereo day images simultaneously. The microphone 16 is installed on the left and right sides of the head 13 in a directional manner toward the front. The left and right microphone cores, 16b, are respectively shown in Figure 1 and Figure 2. That is, it is mounted on the inside of the segment portions 14a and ⑷ arranged on both sides of the envelope M outside the head portion 13. Further, each = 16 :, 16b is through a through hole provided in the segment portion W, 1: =: At the same time, in order to not receive outsourcing "internal tearing," appropriate means are used to give the Duanbu 14a, 14b Xisun π 7 丨 〆7 /, Tou Qiu Fang. The segments 14a and 14b are formed on the segments 14a and 14b so as to face the teeth. The two microphones 16a and 16b can also be used to form a so-called binaural (-) microphone u; it becomes: the outer packaging of the installation position near the microphone 16a is also formed into the shape of a human outer ear. Here, the microphone-pair of the internal sounds collected by the microphone disposed on the inside of the outer package 14 may be based on the noise generated by the internal sound. / Spear, send people's vision inside Jiyiren 10 == Including the above camera 15 and Mike ... The machine system 1 7 is visually and audibly heard by Hefeng branch humans. ^ See group 20, face module 30, stand-up grip钿 y Motor control module 40 and person h · · Limb group] 7. Here, the treatment is made up of-(cuckoo model, group 50), and the batch, 0 mouth group 50 is used as the client Q 丨 彳 f, :: The server is composed of the processing of the server's other modules, that is, ^ relative to each client system and the motor &quot;: Face module 30, three-dimensional model 40. The server and the client are mutually 334432 39 200411627 The Ganshang action is performed in a synchronized manner. Among them, the above server and each client are made up of a personal account, and each of the aforementioned people The computer system is, for example, a communication environment of TCWIP Xiehong rt · · 1 (tiansmission control protocol / internet protocol, value 1 call control protocol / Internet protocol), and the mutual configuration is 1, 8 N (local area network (local area network). At this time, the event of larger data H or Data stream

, Τ 'applies high-speed' 罔 ^ that can perform gigabit data exchange. It is advisable to use the Internet in the robotic audiovisual system 17. In addition, in order to enter Β ^-$ ή Γ ^ Ί -it. ^ ^ V For the temple's control communication, it is best to apply a medium-speed Internet to treat the I-free system 17 as appropriate. As described above, by transmitting a large amount of data at a high speed between individual computers, it is possible to improve the eal time and scaiabiHty of the entire robot. ^ In addition, each module 20, 30, 37, 40, and 50 are hierarchically dispersed to form a 'specifically' from the lower level to the dev layer, the process layer, the feature layer, and the feature layer. The event layer (event) is composed of the above-mentioned auditory module 20 including: a microphone M as a device layer; a peak extraction section 21 as a process layer, a sound source localization section M, a sound source separation section 23, and an active direction pass type filter The device 23 is a pitch 24 as a feature layer (data), a sound source horizontal 25, an auditory event generation section 26 as an event layer, and a speech recognition section 27 and a dialog section 28 as a process layer. Figure 5 works as shown. Also for example, at 48kHz, 16 here, the auditory module 20 is the first, in Figure 5, the auditory module 20314412 20 200411627 bit sampling (sampling) from the microphone ι6 sound plastic; The symbol X1 indicates that the frequency analysis is performed by FFT (FastFouierTrai: sf: r ;;, '= • Fourier transform). If it is expressed by the symbol χ2, the spectrum is generated for each left and right channel. Next, the listening board group 20 extracts a series of half-values' for each of the left and right channels by the peak extracting section 21 to pair the same or similar peaks in pairs for the left and right channels. Peak extraction refers to the use of only high-frequency bands that satisfy (a) the power exceeds the threshold, and (stone) is the local peak, and (τ) is used to cut off (low) low-frequency noise and low power. The frequency band of 90Hz to 3kHz and other two bands (α to 7 ') of the data band filter. The threshold is defined as a value for measuring surrounding background noise and adding a sensitivity parameter such as 10 d] B. Then, the auditory module 20 uses the characteristic that each peak has a harmonic structure to separate the sound sources. Specifically, the sound source separation section 23 extracts the local peaks having a side wave structure according to the order of the frequencies, and regards the set of the extracted peaks as one sound. In this way, the sound signal of each sound source is separated from the mixed sound. When the source is separated, the sound source localization unit 22 of the hearing module 20 is the same as the symbol X 3. The sound source separation unit 23 pairs the sound signals of each sound source that has been separated. ^ The left and right channels select the sound of the same frequency. Signal to calculate IPD (difference from each other) and IID (difference from each other). The calculation is performed, for example, every 5 degrees. Then, the sound source localization section 22 outputs the calculation result to the active direction pass type filter 23a. In contrast, the active-direction pass-through filter 23a is in the combination module 50. According to the calculated direction Θ of the combined data stream 59, as indicated by the symbol X4 2] 314412 200411627, the theoretical value of IPD ipd (= A 0, (0)), and calculate the theoretical value of IID IID (= / \ p '((9)). Among them, direction 0 is based on face positioning (face event 39), stereo vision (stereoscopic event 39a) ) And the sound source localization (auditory event 29), and the result calculated by combining the real-time tracking (symbol X3 ') in the module 50. Here, the theoretical value IPD and the theoretical value are called differently. Mingzhilan. JJJEpipolar Geometry), specifically, set the front face of the robot 10 to 0 degrees, and calculate the theoretical value IPD and theoretical value IID within the range of ± 90 degrees. Here, the above cover is necessary without using HRTF and in order to obtain the direction information of the sound source. In the study of stereo vision, epipolar geometry is one of the most common :: bit methods. Hearing epipolar geometry is the response to epipolar geometry in vision. Next, the audible epipolar geometry system uses geometrical relationships to serve direction information. Therefore, HRTF may not be needed. In the above geometry of the auditory epipolar line, it is assumed that the sound source is infinite, with cry, f, v as 1PD, sound source direction, frequency, and sound speed, respectively. In the case where the human part is regarded as a sphere, the following formula (1) is used when the radius is 0. d) On the other hand, the calculation is performed by using the spectrum of △ 0 and IIDZ \ p. FFT (Fast Fourier Transform) gives the following formulas (2), (3) ιρβ 3M4I2 22 200411627 Δ arctan profile for each frequency band], HSPii V = 201〇g] 0 -arctan ΓMSPrl

wJ% L ⑶ Here, Spl and Spr are the frequency spectrums obtained from the left and right microphones at '16b' at a certain time. Repeatedly, the active-direction pass-through filter 23a selects the passband of the active-direction pass-through filter 23a corresponding to Θ s from the aforementioned data stream direction Θ S according to the pass-band function not represented by the symbol χ7. Here, the 'pass band function system is shown as X7 in Fig. 5', since the sensitivity is maximum in the front direction of the robot (Θ = 0 degrees), the sensitivity can be seen on the side, and it can be known at The minimum value is taken as a large function on the side: This is for the hearing characteristic that the sensitivity will decrease when the left and right angles increase when the sensitivity of positioning in the front direction is the highest. When the positional sensitivity in the frontal direction is the largest, the central small fossa with the visible eye structure of the eyebrows is called the auditory center 2. Regarding the auditory center 1 ', if it is a human, the frontal position is sensitive Degree 'is about 8 degrees around 90 degrees left and right. Δ active direction pass filter 23a uses the selected pass band two to extract the acoustic signal in the range from Θ L to 0 H. Among them, it is defined as eH = 0s + (5 (0s). Active The pass-through filter 2 3 a is as shown in Fumeng χ 5-: Head Transfer Function (HRTF) using the center of the data flow to derive: IPD in ^ and Θ 及 and the theoretical value of D D ㈣ and 314412 23 200411627 ID (△; 0 H (0 s)), that is, the direction of the sound that should be extracted / original. Then, the main 勡 direction pass type filter 23a pairs the sound with the day / original direction 0 according to the auditory nucleus

, Quan Chenghe calculated the IP for each frequency band

η Μ,, Λ Bi E (0)) and IID (= AP ε (Θ)) and Λ n, Μ — △ 0 Η (θ)) and IID (two A h (β)) obtained from hrtf as symbol As shown in X6, the passing band of the right arrow, Wu and Mu Zai, and the pass band 5 (θ) = within the range from the angle to the angle ", the remuneration extracted = △ h) and Π, Λε. The sub-band of the following conditions: quasi =, rfth uses ipd * iid as the judgment base of the crossing wave = cabinet value 'means the upper limit of the effective frequency of the 1PD t positioning system. In the middle, the frequency fth is the wheat branch of the robot 10 'The distance between Yu Shi's gram wind depends on each other, Fang; in the original form, for example, about 〖500HZ. That is, / &lt; Λ ^ Ρη (βι) &lt; Αρ ^ {Qh)

This means that when the frequency is less than the predetermined frequency Q, when there is IPD (back) in the range of HRTF's IPD (common band), and generous; the frequency of the prefetched frequency fth exists in the range of hr. The pass band δ (0) EK Δ p of the taxi p, that is, the collection sub-band is performed. Generally, the impact of IPD in the low frequency band is greater than & i 丄 θ. The IID contention in the frequency band is greater. The threshold frequency f ~~ 1 is dependent on the distance between U and the microphone. Or, the active direction pass filter 23a uses the secondary frequency band collected from the above to eliminate the sound plasticity. The θ sound θ5 can be re-synthesized and constructed into a waveform. The secondary phase work is shown as the sign No. X8.玍 逋 Late person frequency ▼ (Pws-subband) direction, such as the symbol 314412 24 200411627 said X 9 is not the name ^ ._ Shi book, cup ^ 7 person frequency, filtering, as the symbol X10 does not use Inverse frequency conversion IFFT (inverse Fast Fourious nsfoim 'inverse fast Fourier nsfoim'), as shown by the symbol γη, extracts the knife_tone (acoustic signal) from the source of 夂 立 夕 八 σσ in the range. As shown in Fig. 5, the voice recognition unit η is composed of a self-sound control unit 2Ja and an automatic recognition unit 27b. The voice control report "7" is included in the sound signals of the auditory frame, the sound source positioning and the sound source separation, and the sound emitted by the speakers of the dialogue unit 28 described below is excluded, and /, Those who come out with an external acoustic signal. As shown in FIG. 6, the automatic recognition Deng 27b is composed of a speech recognition engine 28, an acoustic model 27d, and a selector 276. For the speech recognition engine 27c, for example, a name developed by Kyoto University can be used. It is the speech recognition engine of "Jurian", which can recognize the vocabulary sent by each speaker. The "automatic recognition unit 26b" shown in Fig. 6 is composed of three speakers such as two males (residents A and C) and one female (speaker B). Therefore, in the automatic identification unit 2 ", an acoustic model 27d is provided for each direction of each speaker. In the case of Fig. 6, the acoustic model 27d is about three speakers a, B, and c respectively. A combination of the voices and directions from each child is provided, and it has a plurality of types. In this case, there are 9 types of acoustic models 27d. °° The edge recognition engine 27c executes 9 speech recognition processes in parallel. At this time, the above 9 acoustic models 27d are used. Specifically, the speech recognition engine 27c uses the above-mentioned nine acoustic models 27d for the sound signals input in parallel with each other to perform the speech recognition process. Then, the results of the above-mentioned words 314412 25 200411627, ie, δ tolerance, are output to the selected photography king ^ Selection 27e. All the above choices cry 27 sound model 27d all μ Xiang. . 27e combines the results of the recognition process with D ° θ from each, ^ ^ +, for example, 可靠性 judges the reliability by a majority decision and hunts down its speech recognition results. ㈣Process result ’and output Here’ ’s the use of the presenter ^ 〆, and the spoon of the horse to make a speech 4 inches a month to determine the voice of the speaker 2 only the zigzag recognition rate. The B-nucleus type and the antelope% are from Bai Baixian. In a 3m x 3m room, 3 people are placed at a distance of 10 meters from the machine, and the direction is 60 degrees from the machine. Secondly, from the speaker output, 150 males, 2 females, and 1 females, such as color, numbers, food, etc., respectively, use the microphones 16a and 16b of the mobile person 10 to collect sounds. Voice data. In #, in the collection of each vocabulary, when 'vocabulary' is used, each vocabulary sounds from only one speaker, two sounds from the same device, and three sounds from the same time. Recording of sound patterns. Then, the voice signal of the recorded voice signal is divided by the active direction pass filter 23a, and each voice data is extracted, and each speaker and direction are sorted to form a training set of the acoustic model. ). Next, in each acoustic model 27d, a triphone is used, and a HTK (Hidden Marcov Mode and Hidden Marcov Model) tool kit 27f is used for each training set to In each direction, a total of 9 types of speech recognition materials are produced. Using the voice data for the acoustic model obtained in the above manner, the vocabulary recognition rate of the acoustic model 27d of a specific speaker is checked through experiments'. The results are shown in FIG. 7. Figure 7 shows the direction on the horizontal axis and the vertical axis 26 314412 200411627: a diagram showing the recognition rate of the capsule, the symbol P represents the sound of the person (speaker A), and the symbol Q represents the other person (speaker B, c) the sound of. In the acoustic model of Talking A, when Speaker A is located on the front side of the robot 10 (pictured), "80% or more vocabulary recognition rate" can be formed on the front side (0 degrees), and Speaker A is located 60 degrees to the right or At 60 degrees on the left, as shown in Figure B or Figure 7 (C), respectively, it can be seen that compared with the speaker's @@, the difference in direction caused by the difference in direction is less, especially Recognition rate of vocabulary above 80%. In considering the result, when the speech recognition is performed, the direction of the sound source is a known point B, and the selectors 27e Α, 隹 一 p 人 甲 / e are used to integrate the cost function V provided by the following formula (5) (pJ. Coffee) Tun 4, Yang ^ +? Like). Like) _like)) Coffee) ^ d) = \ l if K ^ d) ^ Kcs (PeM ,, 1. if ^ s (piCi) ^ Kes ^ Peidj (5) Here, 'V (p, d), Res (p, phantom are defined as the vocabulary recognition rate when using the acoustic model of speaker p and direction d and the recognition of the input voice,'. Fruit, And set de as the sound source direction of real time tracking, and set pe as the speaker to be evaluated. The above Pv (pe, de) is the probability generated by the face recognition module, and the face cannot be recognized It is generally set to 10. Next, the selector 27e rotates the speaker Pe with the largest cost function v (Pe) and the recognition result Res (p, d). At this time, since the selector 27e refers to From the facial module 30, the method of facial event 39 obtained by face recognition 'can be used to specify the speaker, so the robustness of speech recognition can be improved. 3) 44) 2 27 20041 1627 Among them, when the second largest value of the cost function v (Pe) is determined by the speech recognition value being less than 1.0 or close to the candidate, it can be determined that it cannot be defeated or cannot be collected in a pair. Ding Yiyin recognized it, and typed its content &lt; &lt; ^ 诂 诂. 卩 28. The above-mentioned pair of masters, and the speech synthesis unit 2 flutter and cry 28 ° 28 by the dialogue control unit 2Sa… by the later described Combining with heart 6 (^). The speech control result T system of the speech recognition unit 27 of the dialogue control unit is based on the language

Res (p, d) ^ rff) M 4 ^ &amp; ', that is, the speaker Pe and the recognition result VP?}, And the target bricks are generated to the speech synthesis unit 28b. The voice data of the above-mentioned mad stand ^ is output and driven by the voice data called by the control department; the voice corresponding to the Yangli dialect data. Hereby, Fang Z 8e is sent out and the speech recognition unit 27 bis. The speech unit 28 is based on the result of speech recognition. For example, the speaker A says "[] as the number of my favorite robot." .. ^ If the speaker A has 4 make-ups, the speaker A can make a sound of "Ganshi ^^ Brother 1". When the output of the Wuli 1 dialogue unit 28 cannot be identified from the speech recognition unit 27, the robot 10 will leave the speaker A and ask "Are you" yes "to the speaker A? 2? Yes, the answer is again speech recognition. At this time, because the robot 10: 2 and 2 A, the accuracy of speech recognition can be further improved. By this way, the auditory module 20 according to The microphone worker 6 listens to the speaker and extracts his auditory events, and uses the Internet to deal with people, # &quot; ,,, .... The chess group 50 enters the message, and each speaker is identified by voice The dialogue department confirms the speech recognition result to the speaker based on the speech. 314412 28 200411627 Here, in fact, because the sound source is lasting, in order to continue Gushan 4 shi — ☆ function of Caima t, sex, not, such as The source must be considered in the continuous direction of time. As mentioned above, the source direction is obtained based on the real-time tracking. In this way, Fang Shishizhi captures 0 s and calls the event data ... That is,% tracking expresses the consideration of all Ran content as Temporal flow The performance, several sound sources' or even the sound source or the robot itself can be moved in a wide way! Fu Gong-a _ shell material stream can be continuous :: Information. Moreover, times # &amp; + Fang Ran of the source In addition, Bei 4 stream is also used in the integrated vision can refer to the facial events, and see the events, so the way to mention Gu Li Feng a ~ See the sound source positioning accuracy of the wooden branch A.. The above facial module 30 is used as a device layer.

Ml 'face recognition section 32, the face is composed of the face ID34 of the 4-inch sign layer (data), and the face event generation face 36 of the face event layer. In this way, the 3 formulas and 5 styles are taken from the camera according to the group 30; 'the buckle part is changed according to, for example, the skin color extraction, and the two face detection parts of each speaker are obtained.' The other part 32 is based on the previously registered face information = test. When the face is recognized by the face, its face is determined. And, 隹 Jin = Search. When there is a photo, it is determined by the face positioning unit 33. Face direction 35. Here, the face module 30 finds a plurality of materials on the face, and each R locates and tracks according to the image signal. At this time, the face is found by the face; two, the direction and brightness are often Changes will occur, so the size of the part and the detection of the face area will be based on the combination of skin color extraction and phase difference; 31 will be different from the shape of the shoulder 3J4432 29 200411627 pattern matching in 200m seconds Multiple faces can be correctly detected within. The shame &quot; bit position 3 3 transforms the face position of the two-dimensional image plane into a two-dimensional space, and then uses azimuth 0, height 0, and distance! Combination to 'find a face position in one_renyuan space. Then, the face module 3 0 is composed of faces The event generation unit 36 generates a face event 39 for each face from the face ID (name) 34 and the face direction 35, and sends a message to the combination module 50 through the Internet., ', ... 叩From F 丨 Yiyi ’s camera 15, the parallax day image generation section 37a, which is the over-layer, the target extraction section 37b, the target direction 37e 'as the feature layer (data), and the stereo as the event layer : Piece generation # 37d. In this way, the “stereoscopic module 37” is based on: daylight image signal of self-shot ㈣15, and the parallax daylight image generation unit W is used to generate parallax book image from the daylight image signals of both photographers 5 Next, the target extraction unit 37b performs regional division of the parallax day image. When a longitudinally elongated object is found, the target is Η I extraction unit 37b is about to extract it = as a candidate, and determines (locates) it. The target direction 37c. The generating unit generates a three-dimensional event I according to the target direction 37c and sends a message to the combination module 5 through the Internet. Fortunately, the above-mentioned motor control module 40 is composed of a potentiometer (potenti 作为 eter) 42 as a process garment / Motor 41 width modulation, pulse Modulation) control circuit ^, ganga 1 ^ digita analogy digits) conversion circuit 44 and (nalog t〇 ^ for the robot's feature layer of data ❾ 46 $ force machine control section 45, as' and as the event layer of electricity 3] 4412 30 200411627 Motivation event generating unit 47 ^-Group 40, Φ Cheng. In this way, in the electric; dr Y book motive control unit 4 motor control module-described later) instructions, through / K data from the attention Control module 57 (Please drive control. The control circuit 43 supplies the motor 4]. This tree inspection result is transmitted through the rotation control unit 45 of the second detection motor 4]. Next, the motor ㈣ 44 transmits the signal received by the motor control to extract the machine: 豕 by the cent conversion circuit 44 is based on the robot direction: two-way 46. Motor event generation unit 47 Motor event 48, 1 is composed of motor direction information. … Internet and the combination module 5. The above-mentioned combination module 50 is for the module 30, the three-dimensional module 37, the auditory module 20, the face chess group 37, and the motor control module located at the upper position, and constitutes each module M 2G, 3 () ,, 37, 4f) layered data stream layer. The event layer makes the non- "7 from the auditory module", face module 3 :, two = machine control module 'group 40, and the electric &amp; facial event 39, 立 _ 事 # 39 / step, even if Auditory event generating resources: + &amp; and motor event heart synchronization and absolute face conversion section 52 of the two-face data stream 54 and the stereo vision data stream 55; the toilet data is combined with the data stream 59, or I: I: 54, 55 related production Feiyu will release the relationship with the above-mentioned material flow 53, 54, 55 # 56; Note the control module η and the viewer (viewei,) 58o The above-mentioned absolute coordinate conversion unit 52 Synchronize the auditory event 29 from the hearing module 20, the facial event 39 from the face module 30, the stereo event 39a from the stereo module 200411627 37, and the machine event 48 to synchronize, and the machine control module 40 The electric 39 and the stereo event 39a are the same as: See event 29, the facial event is used to convert its coordinate system to an absolute exercise / step motor event 48, which facilitates the facial data stream ... stereo vision: and generates auditory data Stream 53, coordinate conversion _ 52 is the same by flirting "At this time, the above absolute Data stream and stereo vision data stream ^ sensation data stream, stream 53, facial data stream 54 and stereo vision _ = raw auditory data, and the related department 56 is based on auditory resources 54 and stereo vision data stream 55.上 '二: 4 3. The time gate of the facial data flow 丨 Fat skin ten t The day-to-day sequence of the convulsive material flow 53,54,55 on the heart to generate the combined data with the data abortion 55 & or The association is released, and the trembling stream 59 is connected. At the same time, the auditory data aq ^ constitutes the combined data stream 59. See 枓 * 53, the facial data stream 54 is connected to the connection error B1, and the visual lean stream 55 is established. That is, if the father is weak, the relationship will be lifted. The target speaker moves this way, and even if it is ^ LiShou ', it can predict that the speech will go to M 4 to become the corner of its range of movement * Private movement, if it is 53, 54, 55: Fang: then you can generate the above Data stream. 'And to predict the movement of the brothers and sisters in the same day, the left-hand control module 57 is a controller that pays attention to the control of the driving force of the module 40 and combines the two systems of data. At the time, the sequence of 'L auditory data stream 53, face f_visual data &amp; 55's order sample couple went to Shibabei W54 and three-dimensional control mode "and 57 household handle missing, &amp; first' test Pay attention to control. Phonetic… And 57 is based on the auditory data stream 53,-stereo vision data, the state of the dysprosium bismuth stream 54 and the "L" and the presence or absence of the combination module 59, 314412 32 200411627 the last person] [Motion meter.4], the motor event ST motive control module 4 is required to drive the motor command through the Internet. Teleportation as action attention control is based on continuous and 57 households, births, and triggers (iri children remain in the same state, by), jealousy of continuous performance, and Guangxi carries out, for example, delays the most interested objects. Select the data stream that you should pay attention to for tracking. Born into the chase control module 5 7 to perform &amp; type I, &gt; the idea to move the motor ㈣: = the control plan of the branch 4 1 ⑻ ing), the root machine instruction 64a, Η, #, JEL y — seat bag Move 40. The dream is sent to the motor control module 2 and the motor control module 4 via the Internet. According to the motor: 2 '3' PWM control is performed by the motor control unit 45, the driving motor ^ is rotated and driven, and the robot 1G is directed to a predetermined direction ^ The inspector 58 refers to each data stream M to be generated in this way. "", 55, 57 are displayed on the daytime surface of the server. Specifically, chartr chart 58a shows the status of its instantaneous data stream, and it can display the camera's field of view and sound source in more detail. Orientation, the data flow diagram is a combination of data flow (thick line icon) and auditory data flow, facial data flow, and stereo vision data flow (thin line icon). ^ The humanoid robot of the embodiment of the present invention 10 The structure is as described above, and its operation is as follows. First, the distance from the front im of the robot 10 to the left oblique (Θ 460 degrees), the front (Θ 4 degrees), and the right oblique (θι60 degrees) are respectively. The speakers are arranged in a direction, and the robot 10 asks three speakers through the dialogue unit 28, and each speaker answers the questions at the same time. The robot 10 is received by the microphone 16.

JJ 314412 200411627 gathers the speaker's voice &amp; ^ ^, ear, and the auditory module 20 generates auditory 搴 彳 φ accompanying the sound source. π ° dry 29, and transmitted to the combination module 50 through the Internet. To get this way, sheren # ζ one said. The pull group 50 generates auditory failure based on the auditory event 29. The F module 30 reads the golden face of the face obtained by the camera 15 and the resident's image, to generate a facial event 39, and uses the facial data

The speaker's face; 隹 / 一 4,, 'Order and perform face recognition, and at the same time, send the rabbit μ, + 示 # as the result of the face ID24 and the day image to the Internet through 70 People fork group 50. J ： φ ， 木 # _ ，， ° 口 ”s The face of the speaker is not registered in the face data. The branch group 30 will integrate its content into the module 50 through the Internet. Therefore,钍 八 #. Therefore, the combination group 50 is based on the above-mentioned auditory event 29, the Ministry of Knowledge event 39, and the stand-up announcement event 39a to generate a combined data stream π. A, use the two-sense module 2 here. 〇Which pass through the active direction through the type of wave waver? Use the IPD obtained from the geometry of the auditory epipolar line to perform each sound source_ Then # ^ position and separation 'to take out the separation sound (sound signal). The funeral engine π &amp; engine 27 is identified by the "one" recognition unit 29 using voice to identify C, each speaker X, Y, and five standings, and turns to the dialogue unit 28. Dream, say the results in eight ways. The dialogue unit 28 is in the state where the robot 10 is facing the speaker,. ^ ^ Is the aforementioned τ obtained by the voice recognition by the voice recognition unit 27. When the speech recognition unit 27 is performing speech recognition correctly, the machine crying 7 cannot be eclipsed—person 10 is facing the speaker

Next, ask the question again and again, and then repeat the test ^ L according to the answer to voice recognition again. In this way, according to the inventor's humanoid robot 10 and the known humanoid robot 10, according to the separation sound of the sound source position and sound source separation by the hearing module 20 (sound 314412 34 200411627: 11 \ 由The four-tone recognition unit 27 uses a ringing type corresponding to each speaker and direction to perform voice recognition, which can recognize voices of simultaneous &quot; speakers' voices. The multi-digit price of the ground. To the voice recognition unit The action of 27 is evaluated by the user, "As shown in Figure 8, at the distance from the front of the robot 10, the left oblique (θ = +60 sounds), the down and M lead right oblique (0 = -60 sounds)士 Α)) Frontal (θ = 0 degrees), and in the experiment, respectively ^ Speakers X, Ζ, and Z. Among them,; in front of ㈣ talking photo. Here, ', = / 者' at the same time when the acoustic model The same b Yang corpus lineage uses and produces the voices of the speakers in the film. Mao's voice is regarded as a mourning: 妾 妾 'according to the following dialogue version (Scen Yiqi Yan.) Enter the 音 sound recognition 2 machine 10 pairs of three speakers χ talk to Y 〇〇γ, z answer the question at the same time three 仞 from ~, z &amp; Problems. 3 · Stroke ... a y⑼7¾. ^ 〇σ 1 〇 According to the three speakers X, Υ, ζ, 八, 二, 源, 源, 八, 5%, 叱 5 voice, enter; 4 4 丨 Original separation ', and about the separation of each stand, warehouse /. Robot 10 skin-knife away from q for speech recognition. ◦ The speakers X, γ, 7 I were facing each other in sequence. Answer: 2 in the state of returning: 5. When the robot 〖〇 Judging ^ and ", and the method is correct, and the five stand-ups are repeated by the speaker. When you say ^ tolerance, you are being recognized by the sound. The U-Heavy again carried out the Botu diagram. The first example of the results passed by the above version of Hu Jin is shown in ‘3] 44] 2 35 200411627. 由 Ask the person“ What is your favorite number? ” (Refer to Figure 9 (a)) 2. The speakers X, Υ, and Z are the speakers, and the next day will win! Among the numbers up to 10, the sound of reading any number is exiled. For example, as shown in Fig. 9 (b), speaker X says "2", speaker γ says "丨", and speaker Z says "3". 3. The robot 10 uses the microphone 16 to collect sound from the acoustic module 20, and uses the active-direction pass-through filter to perform sound: original position and sound source separation, and extracts the separated sound. Then, based on the separated sounds corresponding to each of the speakers Y z, the speech recognition unit 27 simultaneously executes the speech recognition process using nine acoustic models and performs speech recognition for each speaker. 4. At this time, the selector 27e of the speech recognition unit 27 evaluates the speech recognition assuming that the front side is the speaker γ (Fig. 9 (e)), and then evaluates the speech recognition assuming that the front side is the S-speaker X ( (Figure 9⑷), and finally assume that speaker Z is the front to evaluate speech recognition (Figure 9 (e)). 5 · Then 'selector 27e integrates voice-—0 results into 仃, as shown in Figure 9 (f), regarding the front of the robot (Θ = 〇 户,. ^ Η. ^ 1 degree), decides the most suitable speech The name of the person (Υ) and the result of speech recognition ("i",-; Li Ji is unique to the dialogue department 2 8. In this way, as shown in Figure 9 (g), the machine Y will answer "Y-Kun Is "i" / facing the speaker 6 · Next, regarding the direction of the left oblique (θ = + 60 degrees) &amp; n ^. M] direction, perform the same processing as above, as shown in Figure 9 (h) It shows that the machine cries y ^ °° In the face of the speaker X, the person 10 will answer "χ 君 是『 2 丹 考, regarding the direction of the right oblique (&lt; 9 3144J2 36 200411627 = -60 degrees). The same process is performed, as in the ninth case, the robot 10 will answer the figure B in the state facing the speaker z. At this time, the robot 10 can completely recognize the arrears of each of the speakers VX and B by voice recognition. Therefore, even in the case of simultaneous sending ::, the robot's audiovisual system using the microphone 16 of the robot 10: two middle 'also indicates the sound source localization, sound source separation, and speech recognition. Among them, as shown in Fig. 9 (a), when the robot 10 is not directly facing each speaker, it is like "γjun is" 1 ". Xjun is" 2 ". Zjun is" 3 "口 ° The total number of brothers is "6". "As shown in the figure, it is possible to make the total number of answers =, each speaker =. Fig. 10 shows the second example of the experimental results obtained from the above version. 0 1 · Same as the first example shown in Fig. 9, the robot worker asks "what is the number of straight joy?" (Refer to Fig. 10) (a)). As the speakers X,, Υ, and Z are speakers, as shown in Fig. 10 (b), the speaker X is "2" and the speaker γ is "1". Then, the speaker z is "3 voices." 2. Similarly, the robot 10 performs sound source localization and sound source in the hearing module 20 based on the acoustic signals collected by its microphone through the active direction pass filter 23a. Separate and extract the separation sound, according to the separation sounds corresponding to each of the speakers X, γ, and z, according to each speaker, the 9 recognition unit 27 uses 9 acoustic models to simultaneously execute the voice recognition king and perform its Speech recognition. At this time, the selection of the speech recognition unit 27 37 314412 200411627 2:27: As shown in Fig. 10 (c) ', regarding the positive speaker Y, the b S recognition can be correctly evaluated. 3. Contrary to this, regarding the position of + 60 °, ..1Λ /, the 5 tongue person X 'selector 27e 荇, as shown in the figure 10, cannot be determined to be "2" or "4". ^ 4. = Then, as shown in figure 6), the robot 10 will directly ask the speaker who is located at the degree "Yes" 2 "? Is it" 4 "? / Person position, + 6〇5 · In contrast, as the first! 〇 mgang — Grey Cry Pushshan's answer "Will start from speaker X Zhiyang :: 10 2". At this time, 'speaker x is located on the front of the machine, so the hearing module 20 will answer the speaker's answer to locate the sound source. The sound source is separated, and the speech recognition unit 27 does the speech recognition, and the speaker name X The Japanese language "2" is output to the dialogue unit 28. In this way, as shown in the j ° result chart, the speaker X will be answered "χϋ〇 系 为 第 10⑷ 6 · Next, regarding the speech 纟 ζ system, the reasoning is performed, ^ 1 0 is facing the situation In the state of the living zeta, the answer "Zjun is" 3 Talkers are fine X this; Formula z: Yiren 1〇 By re-inquiring, the speech recognition is correctly performed for each of the sacrifices ^ ... So it shows ^ 1G positive For the side speaker who asked again, the heart of the square = due to: the effect of Fang Zhi's auditory central fossa, the degree of ambiguous cymbal recognition is reduced, which can improve the accuracy of speech recognition and accuracy. Degree 'and can enter =, as shown in the figure 10, the robot 10 correctly enters the sound of each speaker and recognizes 3 ninjas, such as "γjun is"] 疋 xjun is "2". Z 君 is 314412 38 X, 200411627 "3". The bell 1θ "a Μ 疋 b". "As shown in the figure, it is possible to make the total number of the digits Y and Z answered by each speaker. The Le 11 picture shows three examples of experimental results obtained from the above dialogue version. . 10 o (t 1 · Ben: As in the first example shown in Fig. 9, what number is the robot asking me? "(Refer to Fig. 10 (a)). Release as speakers for each speaker χ, γ, and z. Cleaner X is "8", speaker is "", and then the speaker / is the voice of "." Same person as "ground": 0 In the auditory module M, according to the acoustic signal collected by using its microphone 16, reference is made to the data stream direction 0 generated by real-time tracking (refer to χ3,) and the face events of each speaker. The filter 23a performs sound source localization. The sound sources are separated, and the sounds are separated, according to the | | %% — the sounds corresponding to each of the δ brothers X, γ, and z are identified to 5 voices per 5 brothers. Department 27 uses 9 acoustic models to perform the speech recognition process and perform its speech recognition. At this time, 'Voice' recognizes the sound of the selection of the component 2 7 9 θ Β Β °° 27e. Regarding the front speaker Υ, Youji; Yuehankou [3 events judged that the speaker's probability of γ❸ is higher than that of the second point, and when the speech recognition results of various sound models are integrated, (Such as gi◦⑷) No, the situation will be considered. In this way, a more correct language can be performed_ discriminate 5 tolerance. Therefore, as shown in Figure 11 (d), the robot 10 will speak to -X replied "X Jun is" 7 "". 3. Relative to &amp; about speaker X located at +60 degrees, when robot 10 changes direction and faces head-on, about speaker X who is facing front at this time, ^ 412 412 39 200411627 according to facial events for speaker x The probability is high, so similarly, as shown in Fig. 11 (e), the selector 27e will consider the situation. Therefore, as shown in FIG. 11 (f), the robot 10 responds to the speaker x by Γ [gamma] is "8" ". 4. Next, as shown in FIG. 11 (g), the selector 27e performs the same processing on the speaker ζ, and answers the speech recognition result of the speaker z. That is, as shown in Fig. U (h), the robot 10 answers "Z Jun is" 9 "" while facing the speaker z. In this way, the machine and the human 10 are directly opposite each speaker, while referring to their facial events'-identifying according to the speaker's face, and correctly speaking all the answers of each speaker X, Y, and Z. identify. As described above, by identifying the face, it is possible to specify who the speaker is, and thus it is shown that speech recognition with higher precision can be performed. Especially before using it in a specific environment, when face recognition accuracy of approximately 100% is recognized based on face recognition, you can use face recognition information as highly reliable information, and use 2 for voice recognition The number of the audio-visual type 22 used by the speech recognition engine 27c of the unit 27 can be increased, so that higher-speed and high-accuracy speech recognition can be performed. : Ask again. Figure 12 shows the fourth example of the experimental results obtained from the above dialogue version. i The robot 10 asks "What is the favorite fruit?" (refer to Fig. 12 (a)). As shown in Fig. 12 (b), the speaker X speaks the speaker X as the speaker χ, γ, and Z. Is "pear", speaker γ is "watermelon", and speaker Z is "melon". "314412 40 200411627 2 · :: The person 10 is in the auditory module 20, and uses the microphone 16 to perform the SK 5 tiger's source position by the active direction through the type wave filter 23a. The sound source is separated, and Extract the separation sound. Then, based on the separated sounds corresponding to each of the speakers χ, γ, .ζ, each speaker's speech recognition unit 27 uses 9 acoustic models, and simultaneously performs a speech recognition process, and performs its speech recognition. 3. When ==, the selector 27e of the speech recognition # 27 assumes that the front is speaker I and evaluates the speech recognition (calling diagram), and then assumes that the front is the speaker X and evaluates the speech recognition (section 12⑷) Image), after taking it, assume that speaker Z is the front, and evaluate the speech recognition (Figure 12 (e)). 4. Then, as shown in the second figure, the selector 27e integrates the speech recognition results. With regard to the direction of the robot front (θ = 0 degrees), the most suitable speaker name (Υ) and the speech recognition result "Watermelon "And output to the dialogue unit 28. In this way, as shown in Figure 12 (g), the machine crying person 10 will answer 5 in the state facing the speaker Y. Then, each of the speakers X and Z will be phased together. " The processing of the phase question, and answering the speech recognition result for each speech Z. That is, as shown in Fig. L2 (h), 'the robot 10 will face out to the speaker χ, and it will show "χ 君 是「 梨子 」"'. Furthermore, as shown in Fig. 12, the robot 10 Facing "U, I will answer" The Prince is "Hami Melon" ". At this time, the robot 10 can perform correct speech recognition for all rounds of each of the speakers χ, γ, and M. Therefore, the word bag registered by the speech recognition engine We is not limited to numbers. As long as it is a registered word, it can be recognized by 314412 41 200411627. Here, although about 150 vocabularies are registered in the speech recognition bow 2k used in the experiment, the speech recognition rate decreases slightly as the number of syllable syllables increases. ϋ 9

In the above embodiment, the upper body of the robot 10 is composed of 4 earning (degrees of freedom), but it is not limited to this, and the audiovisual system of the present invention may be incorporated in a manner capable of performing arbitrary actions. Posing machine crying. Moreover, in the above-mentioned embodiment, the case where the machine of the present invention is incorporated into the human-type robot 10 is described. I is not limited to this, and various animal-type robots such as a dog-type can obviously be incorporated. Or = other forms of robots. ^ In the above description, the example of the configuration in which the robotic audiovisual system 17 is equipped with the stereo module 37 is shown in the figure. However, the component may not be provided. At this time, the combination module 50 generates the auditory information of each speaker based on the auditory facial event% and the motor module 48, and then generates a combination with # ”facial information ... Module 59 and =: Zuoyi control module 50 is configured to perform Zhuyin control based on the above data stream. ~ Furthermore, in the above description, the active direction pass type filter system, °, and the pass band range (Pass range) control, setting = given target range is constant, regardless of the frequency of the sound being processed. Here, the passband 6 is derived, using 100Hz, 200Hz, 500Hz, 1000Hz, 2000H2 Μ ς The pure sound of the harmonic structural sound () iarrnonics of ， 1 〇〇Ηζ and ~ j 固 xi, shi, shi, and ba yi conducted experiments on the sound source extraction rate of one sound source. Among them, 42 314412 200411627 made the sound source From the position of 0 degrees on the front of the human body to the left or 90 degrees of the main weapon, move the position once every 10 degrees. Figures 13 to 15 show Xi Xi Cong Mai 1 # 班 仕奕 0 ° to 90 ° each position When the sound source is set, the sound source is not inserted. As shown in the experimental results, according to the frequency of the makeup frequency band, the pass band frequency is phase-locked, n cesium, and the system is used to improve the extraction rate of sound with special frequencies. Τ The accuracy of the separation of Li Yili. In addition, you can also improve the recognition rate of the sound. Therefore, at .17 φ Fang 'the above-mentioned state of the robot's audiovisual system, washing 17' active direction through type tears cry # 41 ^ The pass band range of 23a can be controlled according to the mother-frequency. [Industrial Utilization Possibility] As described above, according to the present invention, the benefits are better; The wrong reason is to use multiple audio modes ^ if it T ^ ^ x ^ 扪, you can enter the correct voice in real time and real time%, and you can distinguish the sound of the hibiscus and the sound by the selector. In the voice-fork type, my voice recognizes and recognizes the voice recognition results, so as to determine the correct voice recognition with the highest reliability than the conventional voice recognition. [Schematic description] Bi Youshi π Ψ ^ Ψ s Moon's vision Put it first ... The front view of the appearance of the evil humanoid robot.! 2 is a side view of the humanoid robot shown in Figure 1. Brother 3 'is the figure of Brother 1's figure. Figure 4 is a block diagram showing the structure of the system of the circle] ^ 铖 宁 态 态 宁 宁 宁 宁 宁 宁 宁 宁 宁 视 Human hearing and hearing buns structure example block diagram. 334412 43 200411627 Figure 5 shows the role of the spear module Figure 6: Brother L Seeing Listening in the System Figure 6 is a model of the auditory vision in the robot's audio-visual system shown by the voice recognition unit of the watch spear group. Outline of the construction example of the speech recognition engine. Squint left and right two: = Figure 6 shows the front of the speech recognition engine and the _ sign showing the sound of the person who made the sound, and then the speaker who is right oblique. Situation. ^ Lee! : The diagram is a schematic oblique view showing the language in the robot's audiovisual system shown in Figure 4 as a household and an inspection. Figures ⑷ to ⑴ show the results of the first-example of the sound recognition experiment of the robotic audiovisual system of Figure 4 in order: Figures (a) to (i) are diagrams showing the results of the second example of the sound recognition experiment of the robotic audiovisual system of Figure 4 in order: Figures 11 (a) to (h) are diagrams showing the results of the third example of the 2-tone recognition experiment of the robotic audiovisual system of Figure 4 in order. Figures 12 (a) to ⑴ show the results of the fourth example of the speech recognition experiment of the robotic audiovisual system of Figure 4 in order. FIG. 10 is a drawing showing the extraction rate in the case where the active frequency band passing through the j filter has been controlled in accordance with the embodiment of the present invention, and ⑷ is at 0 degrees, (b) is at 10 degrees, When 具有 is at 2P degrees and ⑷ has a sound source in the direction of j 〇 degrees. FIG. 14 is a diagram showing the pass band frequency output rate of the active side 44 314412 2004110 ^ / pass filter related to the embodiment of the present invention, showing (a) When the direction has a sound source. (B) The 5015 graph at 50 degrees and (c) at 60 degrees is a graph showing the output rate of the pass filter, and shows (a) the situation where there is a sound source in the direction. Relevant active side $ frequency ▼ 70 degrees when the range has been controlled '(b) at 80 degrees, (c) at 90 degrees 10 humanoid robot 11 12 carcass 13 13a link member 14 14a, 14b Part 15 16, 16a, 16b Microphone 17 20 Hearing module 21 22 Source localization part 23 23a Active direction pass filter 24 Pitch 25 26 Hearing event generation part 2 7 27a Acoustic control part 27b 2 7c Speech recognition engine 27d 27e Selection 27f 28 dialogue unit 28a 28b vowel synthesis unit 28c 29 auditory event 30 base head outer packaging camera robot audiovisual system peak extraction unit sound source separation unit sound source horizontal direction voice recognition unit automatic recognition unit acoustic model HTK tool suite dialogue control unit Speaker face module 3] 44] 2 45 200411627 3 1 Face detection section 33 Face positioning section 35 Face direction 37 Stereo module 37b Target extraction section 37d Stereo event Part generation part 38a Stereo event® 40 Motor control module 42 Potentiometer 44 AD conversion circuit 46 Robot direction 48 Motor event 52 Absolute coordinate conversion part 54 Face data stream 56 Related part Φ 58 Observer 58b Data flow diagram 64a Motor command 32 Face recognition section 34 Face ID 36 Face event generation section 37a Parallax day image generation section 37c Target direction 38 Face database 39 Face event 41 Motor 43 PWM control circuit 45 Motor control section 47 Motor event generation section 50 Combination mode Group 53 audio data stream 55 stereo vision data stream 57 attention control module 58a radar chart 59 combined data stream 70 internet 46 314412

Claims (1)

200411627 Scope of patent application: i. A robotic audiovisual system, with Λ, ^ A ^ ^ ^, equipped with a combination of the spoken spoon of each speaker and its direction. Acoustic models; using these acoustic models, the speech recognition of the b-day recognition process for the stereo signals that have undergone standing, widening, and opening ^^, +, &amp; A hand, and, for The aforementioned b-recognition process is integrated according to the aforementioned acoustic model and the results of a plurality of voice recognition processes, and the selector _ is accompanied by any one of the sound recognition process results. Among them, each of the speakers is spoken simultaneously Vocabulary for identification. For example, the robotic audio-visual system according to item 1 of the scope of the patent application, in which the Dadatin selector is constituted by a majority decision to select the result of the aforementioned speech recognition process. 3. If the robotic audiovisual system of item 1 or item 2 of the scope of patent application, it is provided with a dialogue unit that outputs the result of the voice recognition process selected by the selector to an external. 4. A robotic audiovisual system, comprising: at least one pair of microphones for collecting external sounds; according to the acoustic signals emitted by the microphones, the sound source is separated by a collection of spectral waves extracted according to the height of the sound And positioning to determine the direction of at least one speaker, and to extract the auditory module of its auditory event; a camera with a camera in front of the robot. According to the day image photographed by the camera, The speaker's face recognition and positioning, identify each speaker, and extract the face module of his face event; 47 314412, 'the robot is rotated in the horizontal direction. According to the driving force of the driving motor, Root control module; set the motor for extracting the motor event from the auditory event, facial event, and audio event location of the facial event and the face of the facial event, and determine the direction of each speaker according to the direction of the filter), , χ s * ^ „Use Kalman filter (Kalman… Ke k and facial data stream, and then connect with Shi to generate a combination of combined data streams Groups, &quot; machine-related attention control based on these streams, and the action plan results from attention control for electric motors :: Attention control module; where:? Mechanism: The auditory module is based on From the aforementioned Π-direction information ', which is the smallest in the front direction, and which becomes larger with the left 2 ^ and A, the active direction pass-through wave passer of the pass range will have two in a predetermined wide range. The sub-bands of inter-ear phase difference (IPD) or inter-ear intensity difference (IID) are collected and the waveforms of sound sources $ ,, * 彡 ^ θ are reconstructed to separate the sound sources, and multiple sound models are used to perform The speech recognition of the audio signals separated by the sound source is combined by the selector to the speech recognition results obtained by the respective acoustic models. It is constituted by judging the most reliable speech recognition result among the speech recognition results. 5. A robotic audiovisual system, comprising: at least one pair of microphones for collecting external sounds, based on 314412 48 200411627音 变 # issued by the wind, the harmonic wave broadcasts the number 1 θ 乜, and the sound source is separated and located according to the height of the sound— + / &quot; &quot; to determine the direction of the speaker. And extract its hearing and module; D.4. Jue has a camera that takes pictures of the front of the robot, and recognizes and locates each speaker's face based on the portrait taken by the camera. According to the parallax extracted from the day image captured by the stereo camera, 'feed and position the goods to a longer object, and extract the three-dimensional event post module; 1 There are two control modules for driving the electric motor that rotates the robot in the horizontal direction according to the rotation position of the driving motor; the motor for extracting the ㈣ event is composed of the aforementioned auditory event, facial event 'stereo event and motor event' according to the auditory event. Sound source positioning and face event orientation information to determine the direction of each speaker. Use a kalmanfiker to connect the aforementioned events in the time direction to the decision. Generate auditory data streams, facial data streams, and stereoscopic data streams, and then combine the above data streams to generate a combined module that combines the data streams; and perform attention control based on these data streams, and The resulting action plan results in an attention control module that performs motor drive control; wherein, the aforementioned auditory module is based on the correct sound source direction information according to 3) 4412 49 from the aforementioned combined module, and it becomes larger when the positive angle becomes larger. Larger: It is a minimum type and it passes through the filter in the left and right direction. It will have a main interphase difference (IPD) of 1 interfering range (ah range) or two ears. ^ Two ears within a predetermined wide range. Set, and reconstruct the sub-band of the sound source ^ (IID) and use multiple sound models to separate / source the sound sources. At the same time, the speech recognition is integrated by the sound signal recognition result of the selector sound source separation. Among the identification results of the 1st oldest temple with the highest reliability judged by 哕 *, * 〖Recognition result of ^ sound retrieved 6 · As for the 4th in the scope of patent application, where m, ' +, The robotic audiovisual system, note that when the “trigger group” is used for speech recognition, the above-mentioned pressing center panel system uses the sound signals of,,, and the camera, and then A ′. Yue Xiangyue talks about the voice of the microphone, and then the microphone collects the sound. The sound source positioning and separation by the auditory module are composed.卞 海 耳 说 进 仃, my voice recognition 7. For example, the scope of the patent application No. 4 苴 Zhongfeidi 5 machine vision audiovisual system; Listening group is doing speech recognition, iBS is obtained from the face neighbor module Facial events. ", The robotic audiovisual system of item 5 of the scope of patent application by deduction β δ · 1 °, in which the group refers to the three-dimensional event served by the three-dimensional module when performing speech recognition. 9 ·: Application for June patent scope The robot visual and auditory system of item 5, wherein when the auditory module performs speech recognition, it refers to the facial events obtained by the facial module 3] 44] 2 50 200411627 and the stereo events obtained by the stereo module. 1 10. If the robot vision and hearing system of item 4 or item 5 of the scope of patent application, _ which includes a dialogue unit that outputs the speech recognition result judged by the aforementioned hearing module to the outside. 1 1. If the patent is applied for The robot audiovisual system of the range item 4 or item 5, wherein the passband range of the aforementioned active direction pass filter can be controlled for each frequency. 51 314412