JP2009080309A - Speech recognition device, speech recognition method, speech recognition program and recording medium in which speech recogntion program is recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect a time period when noise is dominant, in sound source locating process, and to perform suitable processing in this time period in speech recognition process. <P>SOLUTION: Phase difference and power are calculated for each frequency component from first and second sound signals which are captured in two points, and a scatter diagram in which frequency and phase difference for each frequency component are set as coordinates is created. Arrangement of the frequency component for showing predetermined linearity on the scatter diagram is detected, together with a straight line score according to the power of the frequency component. The arrangement in which the straight line score is a predetermined threshold or more, is detected as the straight line for indicating existence of the sound source. A sound source stream composed of information of the straight lines and the straight lines scores etc. is extracted, and reliability information is attached for each time of the sound source stream, based on height of the straight line score for each time of the sound source stream. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an apparatus for recognizing speech, and more particularly to an apparatus for detecting and localizing a target sound source in a noisy environment, and separating and extracting sound data of the target sound source from noise.

  In recent years, in the field of auditory research for robots, the number and direction of multiple target sound sources are estimated in a noisy environment (sound source localization), and the sound from each sound source is separated and extracted (sound source separation). A method for recognition (voice recognition) has been proposed.

  For example, using a pair of microphones, a method of performing localization of a directional sound source (target sound source) and sound source sound under a diffuse noise environment has been proposed (see, for example, Patent Document 1). In this method, a phase difference for each frequency between two acoustic signals is obtained from frequency-resolved data obtained by Fourier transforming two acoustic signals captured by two microphones, and this is plotted in a frequency-phase difference coordinate system. Is generated. Considering that frequency components with the same arrival time difference are derived from the same sound source, pay attention to the fact that these frequency components are distributed on a straight line passing through the origin on the scatter diagram. Detecting the sound source and locating the direction by performing straight line detection using If two straight lines with a Hough vote vote value (also called a score) exceeding a predetermined threshold are detected, there are two sound sources with different directions, and the direction of each sound source is known from the inclination of each straight line. be able to. Furthermore, the straight line obtained at each time is grouped in time series by paying attention to the slope, and the target sound source stream is formed. The sound from the sound source (target purpose) is formed by beam forming with directivity in the sound source direction. Audio). The separated and extracted speech (separated speech) is recognized as speech, and the linguistic information of the target speech is estimated.

  This prior art first detects the presence of each sound source and estimates its direction (sound source localization) based on the arrival time difference between the microphones (directional sound source = target sound source) emitted from a local region in space. . Then, each sound source sound is separated and extracted (sound source separation) from other sounds (other directional sound source sounds and diffusive environmental noise) by beam forming with directivity in each sound source direction, and this separated sound is recognized ( Voice recognition).

  However, if multiple sound sources are simultaneously producing sounds of the same frequency, the amplitude vector for each frequency obtained by Fourier transform becomes a composite vector of the amplitude vectors of each sound source sound on the complex plane. Finding the phase difference does not represent the correct arrival time difference. That is, such frequency components do not apply to any sound source direction and are lost during beam forming, so that the extracted separated speech is distorted. If the acoustic model used at the time of speech recognition does not learn this distortion, the acoustic score does not increase in the likelihood calculation at the time of speech recognition, resulting in erroneous recognition.

  In order to suppress the influence of this distortion, a speech recognition method is shown that masks the component of the feature quantity affected by the spectral distortion and does not use it for likelihood calculation (for example, see Non-Patent Document 1). At that time, noise estimation is performed in the sound source separation process, and a mask is automatically generated using the estimated noise information. As described in this document, this method is a method of “integrating sound source separation and speech recognition”.

  Even when a person is speaking one phrase, since the sound pressure is strong and weak, the speech is lost to the environmental noise in the weak part, and the extracted separated speech for that period It is impossible to observe. In such a case, for example, a part of the utterance content that should be audible in a quiet environment could not be observed in a noisy environment, so the speech recognition given to the grammar information interprets this utterance content during this period. There is a high risk of failure and misrecognition. In particular, as the noise intensity increases, such a period becomes longer, and the risk increases.

  As an example of detecting and using a change in the strength of speech during utterance, a method of controlling likelihood calculation according to silence likeness in input speech is shown (for example, see Patent Document 2). In the embodiment, there is a description that "... the beam search is used, but the beam width is narrowed down in the silent section", and some of the likelihoods are high at each time. In the beam search for pruning while leaving the above hypothesis, it is described that the amount of recognition processing in the silent section is reduced by reducing the hypotheses left in the silent section. This is because there is no speech information in the silent section in the first place, so that unnecessary calculations during that period are reduced. In addition, as a method for determining whether or not it is a silent section, (A) a method for determining whether or not a period during which the power of the input voice is higher than a predetermined threshold continues, and (B) an acoustic score that is collated with a silent acoustic feature, Two methods of determining whether or not a period higher than an acoustic score collated with an acoustic feature other than silence continues are shown.

  On the other hand, by applying the sound source localization process of Patent Document 1, it is possible to determine that the vote value of the Hough vote increases according to the power of each frequency component, so the magnitude of the vote value for each direction is the sound source sound from that direction. As a result, the period in which the environmental noise is dominant can be detected by a phenomenon that the straight line cannot be detected or the vote value of the straight line is small. For this reason, even when the input sound is dominated by strong noise rather than silence due to strong environmental sound, it can be detected that the sound from a certain sound source tends to be interrupted by the environmental sound. The “silent detection means” of Patent Document 2 does not have this capability.

  In the speech recognition process, it is expected that the process during the period in which such environmental noise is dominant can be adjusted well. This corresponds to an approach of “integrating sound source localization and speech recognition” as opposed to Non-Patent Document 1 that “integrates sound source separation and speech recognition”. The prior art described above does not suggest this integration.

As for the detection of a straight line, a technique for detecting a straight line on a frequency-phase difference scatter diagram is disclosed (for example, see Patent Document 3). However, this method is the same as Patent Document 1 in that it is evaluated assuming straight lines with various inclinations on the scatter diagram, but there is a difference in the method. Patent Document 3 is a method of detecting a straight line that is a least square error with respect to the frequency component arrangement on the scatter diagram, and an evaluation amount (assumed as a sum of squares of the distance between each frequency component on the scatter diagram and the straight line). Therefore, the vote value corresponding to the power as in Patent Document 1 is not obtained. If it is desired to know the dominant period of noise from the vote value, it is more convenient to obtain the vote value as a function of power (or amplitude).
JP 2006-254226 A Japanese Patent Laid-Open No. 11-85180 JP 2003-337164 A Shunichi Yamamoto et al., “Simultaneous speech recognition based on automatic generation of missing feature masks by integration with sound source separation”, Journal of the Robotics Society of Japan, Vol. 25, no. 1. Issued on January 15, 2007

  The present invention has been made in view of the above-mentioned problems and considerations. The object of the present invention is to (1) detect a period in which noise is dominant in a sound source localization process, and (2) It is to provide a voice recognition device, a voice recognition method, a voice recognition program, and a recording medium on which a voice recognition program is recorded, capable of performing recognition while suppressing the adverse effect of noise when recognizing separated sound of a sound source stream. .

  A speech recognition apparatus according to an aspect of the present invention includes an input unit that inputs first and second acoustic signals captured at two points, and frequency-resolves each of the first and second acoustic signals. Calculating means for calculating a phase difference and power for each frequency component; generating means for generating a scatter diagram having the frequency component value and the phase difference value as coordinate values; and on the scatter diagram. Detecting means for detecting the arrangement of frequency components exhibiting linearity together with a linear score corresponding to the power, and detecting the arrangement of frequency components for which the linear score is equal to or greater than a threshold value as a straight line indicating the presence of a sound source; and a certain range A sound source stream that groups at least one straight line detected by the detecting means in a time axis direction while allowing a straight line non-detection period and a straight line inclination fluctuation, and includes information including the straight line inclination, Extraction means for extracting a sound source stream including information about a line score and a time when the straight line is detected; and reliability information based on the level of the straight line score is given to the time of the sound source stream, and the sound source stream Classifying means for classifying each frame of the sound source, sound source separation means for extracting sound data of the sound source stream based on the sound source existence angle calculated from the information of the slope of the straight line included in the sound source stream, and separating the sound source; The sentence hypothesis defined in the grammatical information is expanded into a state and transition search tree, a predetermined acoustic feature is extracted from the sound data of the sound source stream, and the likelihood of the state transition path of the search tree for the acoustic feature sequence And speech recognition means for recognizing the linguistic content of the sound source stream by searching for a state transition path with high likelihood, and the state transition And controlling based on a search of the road to the trust permission information.

  According to the present invention, (1) a period in which noise is dominant is detected in a sound source localization process, and (2) when recognizing separated speech of a target sound source stream, recognition is performed while suppressing adverse effects of noise. Can do.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments of a voice recognition device, a voice recognition method, a voice recognition program, and a recording medium that records the voice recognition program according to the present invention will be described with reference to the drawings.

  FIG. 1 shows functional blocks of a speech recognition apparatus according to an embodiment of the present invention. The speech recognition apparatus according to the present embodiment includes microphones 1a and 1b arranged at spatially different positions, an acoustic signal input unit 2, a sound source stream extraction / classification unit 3, a sound source separation unit 4, and a vocabulary recognition unit 5. A speaker recognition unit 6, a sound recognition unit 7, an output unit 8, and a user interface unit 9.

  Two amplitude data from the microphones 1 a and 1 b are input to the sound source stream extraction / classification unit 3 via the acoustic signal input unit 2. The sound source stream extraction classifying unit 3 first frequency-decomposes a predetermined number (frame length) of amplitude data by (1) FFT processing for each discrete time (frame) repeated at a predetermined time interval (frame shift). (2) The phase difference between both inputs is obtained for each frequency component. At this time, for example, an average of the power values of the frequency components at both inputs is obtained as the representative power value of the frequency components.

  Next, the sound source stream extraction classifying unit 3 (3) converts the phase difference for each frequency of successive predetermined frames into a two-dimensional scatter diagram on the frequency-phase difference plane, and (4) determines a predetermined difference from the two-dimensional scatter diagram. A straight line is detected along with its straight line score. The detected straight line suggests the existence of a certain directional sound source (target sound source). At this time, the frequency component distributed in the vicinity of the straight line approximates the spectrum at the time (frame) of the sound (sound source sound) emitted from the target sound source, and is a straight line calculated based on the representative power value of these frequency components. The score gives a measure of the total power of the sound source. In the present embodiment, this straight line score is calculated by the vote value of the Hough vote. Further, the detected slope θ of the straight line has a one-to-one correspondence with the target sound source existing angle φ (the opening angle of the conical surface where the target sound source exists) with respect to the line segment connecting the microphones 1a and 1b. (5) The straight line no-detection is grouped in the time axis direction while allowing straight line no-detection periods and straight-line tilt fluctuations within a predetermined range on data obtained by arranging straight line inclinations θ on an angle-time plane. A sequence of the detected straight line having a length equal to or longer than a predetermined period including a period, a sequence of the inclination θ and a sequence of the existence angle φ calculated therefrom, and an existence period of the detected series of straight lines (the above-mentioned The period between the start frame and the end frame of the grouping) is detected as information of an audio stream (target sound source stream) emitted from one target sound source.

  In particular, the sound source stream extraction / classification unit 3 according to the present embodiment (6) for each frame sandwiched between the start and end of the target sound source stream, the frame in which the straight line is grouped in the target sound source stream is not reliable. Classify the frame as unreliable. This reliability indicates whether the sound source sound of the target sound source stream is intelligible at each time (each frame).

  The processes (1) to (5) can be performed by the technique disclosed in Patent Document 1.

  The sound source separation unit 4 separates the sound data (target sound data) of the target sound source stream from the environmental noise by performing beam forming on the input sound data based on the existence angle series.

  Through the above processing, audio data of the target sound source stream (referred to as target audio data or separated audio data) and reliability information at each time (frame) are obtained. An unreliable frame is interpreted as representing a period during which a straight line could not be clearly detected because (a) the target voice was defeated by strong environmental noise, or (b) the target voice was originally weak or absent. it can. In particular, in the case of (a), for example, a part of the utterance content that should be audible in a quiet environment could not be observed in a noisy environment. Therefore, in the conventional speech vocabulary recognition given to the grammatical information, In this period, there is a high risk of misinterpretation due to failure to interpret.

  Similarly, in speech speaker recognition and object sound recognition, there is a high risk that speech recognition during a period in which noise is dominant cannot be recognized correctly.

  The vocabulary recognition unit 5 is a means for recognizing the linguistic contents of the target speech data, and uses only frames classified as reliable in the likelihood calculation when calculating the likelihood for interpretation according to the grammatical information. The occurrence of misrecognition is suppressed by not pruning the frames classified as.

  The speaker recognition unit 6 is means for recognizing who the target voice data is, and suppresses the occurrence of misrecognition by recognizing only frames classified as reliable.

  The sound object recognition unit 7 is a means for recognizing what kind of sound the target sound data is, and suppresses the occurrence of misrecognition by recognizing only the frames classified as reliable.

  The output unit 8 generates and outputs sound source information including at least the number of the target sound source streams, the existence angle series of each target sound source stream, and the recognition result obtained by recognizing the target sound data.

  The user interface unit 9 presents various setting values to the user, accepts setting input from the user, saves the setting values to the external storage device, reads the setting values from the external storage device, and users of various processing results Make a presentation to

  Hereinafter, the operation of each functional block of the speech recognition apparatus according to the present embodiment will be described in detail.

(Basic concept of estimating sound source from phase difference for each frequency component)
The microphone 1a and the microphone 1b are two microphones arranged at a predetermined distance in a medium such as air, and convert medium vibrations (sound waves) at two different points into electrical signals (acoustic signals), respectively. It is means of. Hereinafter, when the microphone 1a and the microphone 1b are handled together, they are referred to as “microphone pairs”.

  The acoustic signal input unit 2 periodically A / D-converts two electrical signals (acoustic signals) from the microphone 1a and the microphone 1b at a predetermined sampling frequency Fr, so that the two acoustic signals from the microphone 1a and the microphone 1b are converted. It is a means for generating digitized amplitude data in time series. By analyzing this input amplitude data by decomposing into phase differences for each frequency component, even if multiple sound sources exist at the same time, frequency components peculiar to each sound source are each between two data. Therefore, if the phase difference for each frequency component can be divided into groups with the same direction, there are several sound sources for a wide variety of sound sources, You should be able to figure out what kind of voice each person is making and how much strength or power it is.

(Audio stream extraction and classification unit 3)
FIG. 2 shows an internal configuration of the sound source stream extraction / classification unit 3 that realizes the above basic concept. The sound source stream extraction and classification unit 3 includes a frequency decomposition unit 301, a phase difference calculation unit 302, a scatter diagram generation unit 303, a voting unit 304, a straight line detection unit 305, a time series tracking unit 306, a duration evaluation unit 307, and a frame classification unit 308. Consists of.

(Frequency decomposition unit 301)
The frequency resolving unit 301 receives the amplitude data a and b generated by digitizing the sound signals captured by the microphones 1a and 1b by the sound signal input unit 2, and the frequency resolving data a and b obtained by decomposing each into frequency components. Generate. A general technique for decomposing amplitude data into frequency components is fast Fourier transform (FFT). As a typical algorithm, a Cooley-Turkey DFT algorithm and the like are known.

  The frequency resolving unit 301 extracts N pieces of continuous amplitude data from a certain time (T-th frame) from the amplitude data obtained by the acoustic signal input unit 2 and performs FFT processing, and sets the extracted position to a predetermined frame. It is repeated at discrete times (T + 1 frame, T + 2 frame,...) While shifting by the shift amount Fs. As a result, frequency-resolved data composed of the power value and phase value for each frequency component with respect to the input amplitude data is generated in time series.

(Phase difference calculation unit 302)
The phase difference calculation unit 302 compares the two frequency decomposition data a and b at the same time obtained by the frequency decomposition unit 301, and calculates the difference between both phase values for each same frequency component. Interphase difference data is generated. As shown in FIG. 3, the phase difference ΔPh (fk) of a certain frequency component fk is calculated by calculating the difference between the phase value Ph1 (fk) at the microphone 1a and the phase value Ph2 (fk) at the microphone 1b. (Fk): Calculated as a 2π residue system so that −π <ΔPh (fk) ≦ π}.

(Scatter diagram generator 303)
Based on the inter-ab phase difference data obtained by the phase difference calculation unit 302, the scatter diagram generation unit 303 generates coordinate values for handling a set of frequency and the phase difference as points on a predetermined two-dimensional XY coordinate system. It is a means to determine. The X coordinate value x (fk) and the Y coordinate value y (fk) corresponding to the phase difference ΔPh (fk) of a certain frequency component fk are determined by the equations shown in FIG. The X coordinate value is the phase difference ΔPh (fk), and the Y coordinate value is the frequency component number k. A plot of such point clouds in the XY coordinate system is a scatter diagram.

(Frequency proportionality of phase difference for the same time difference)
The phase difference for each frequency component calculated by the phase difference calculation unit 302 as shown in FIG. 3 should represent the same arrival time difference between those derived from the same sound source (in the same direction). At this time, since the phase value of a certain frequency obtained by FFT and the phase difference between both microphones are values calculated by setting the frequency period to 2π, the phase is also 2 if the frequency is doubled even at the same time. There is a proportional relationship that doubles. The same applies to the phase difference, and the phase difference with respect to the same time difference ΔT increases in proportion to the frequency. When a scatter diagram in which the phase difference of each frequency component emitted from the same sound source and having a common ΔT is plotted on the two-dimensional coordinate system by the coordinate value calculation shown in FIG. 4 is generated, the coordinates representing the phase difference of each frequency component are generated. The dots are arranged in a straight line. The greater the ΔT, that is, the greater the distance to the sound source between the two microphones, the greater the slope of this straight line.

(Circulation of phase difference)
However, the phase difference between the microphones is proportional to the frequency throughout the scatter diagram only when the true phase difference does not deviate from ± π from the lowest frequency to the highest frequency to be analyzed. This condition is that ΔT does not become equal to or longer than the time corresponding to ½ period of the maximum frequency (half the sampling frequency) Fr / 2 [Hz], that is, 1 / Fr [second]. If ΔT is 1 / Fr or more, it must be considered that the phase difference can only be obtained as a cyclic value as described below.

  A phase value for each frequency component that can be obtained can only be obtained as an angle value on a complex coordinate system with a width of 2π [radian] (in this embodiment, a width of 2π between −π and π). This means that even if the actual phase difference in the frequency component is opened for one period or more between both microphones, it cannot be known from the phase value obtained as a result of frequency decomposition. Therefore, in the present embodiment, the phase difference is obtained between −π and π. However, the true phase difference caused by ΔT may be a value obtained by adding or subtracting 2π to the value of the phase difference obtained here, or adding or subtracting 4π or 6π.

  FIG. 5 is a scatter diagram schematically showing this. When the phase difference ΔPh (fk) of the frequency fk is + π as represented by the black circle 140 in the figure, the phase difference of the next higher frequency fk + 1 exceeds + π as represented by the white circle 141 in the figure. However, the calculated phase difference ΔPh (fk + 1) is a value slightly larger than −π as indicated by the black circle 142 in the figure, which is obtained by subtracting 2π from the original phase difference. Further, although not shown in the figure, the same value is shown even at three times the frequency, but this is a value obtained by subtracting 4π from the actual phase difference. Thus, the phase difference circulates between −π and π as a 2π residue system as the frequency increases. As shown in this example, when ΔT increases, the true phase difference represented by a white circle circulates on the opposite side as indicated by the black circle above a certain frequency fk + 1.

  The problem of estimating the number and direction of sound sources in the present invention can be reduced to detecting a straight line as shown on such a scatter diagram. The problem of estimating the approximate frequency component for each sound source can be reduced to selecting the frequency component plotted at a position close to the detected straight line. Therefore, the scatter diagram data output from the scatter diagram generation unit 303 in this embodiment is a point group determined as a function of frequency and phase difference using the frequency decomposition data by the frequency decomposition unit 301. The voting unit 304 detects a linear arrangement as a figure from the point cloud arrangement given as the scatter diagram data.

(Voting section 304)
The voting unit 304 applies a linear Hough transform to each frequency component given the (x, y) coordinates by the scatter diagram generation unit 303 as described later, and the trajectory in the Hough voting space by a predetermined method. A means to vote. Hough conversion is described in Akio Okazaki, “First Image Processing”, Industrial Research Committee, pages 100 to 102, published on October 20, 2000.

(Linear Hough transform)
There are an infinite number of straight lines that can pass through the point p (x, y) on the two-dimensional coordinates, but the inclination from the X axis of the perpendicular drawn from the origin O to each straight line is expressed as θ, and the length of the perpendicular is expressed as ρ. Then, θ and ρ are uniquely determined for one straight line, and a set of θ and ρ that can be taken by a straight line passing through a certain point (x, y) is a locus unique to the value of (x, y) on the θρ coordinate system. It is known to draw (ρ = x cos θ + y sin θ). This locus is called a Hough curve. Further, such a conversion from the (x, y) coordinate value to a locus of (θ, ρ) of a straight line passing therethrough is called a linear Hough transform. Note that θ is positive when the straight line is tilted to the left, 0 when vertical, and a negative value when tilted to the right, and the definition range of θ is {θ: −π <θ ≦ π}. There is no departure.

  The Hough curve can be obtained independently for each point on the XY coordinate system. For example, a straight line passing through the three points p1, p2, and p3 is a point where three trajectories corresponding to p1, p2, and p3 intersect. Can be obtained as a straight line defined by the coordinates (θ0, ρ0). The more lines that pass through the points, the more trajectories pass through the positions of θ and ρ representing the lines.

(Hough voting)
In order to detect a straight line from a point cloud, a method called Hough voting is used. This is because by voting a set of θ and ρ through which each trajectory passes in a two-dimensional Hough voting space with θ and ρ as coordinate axes, θ and ρ through which a large number of trajectories pass at a large position in the Hough voting space. This is a technique for suggesting the existence of a pair, that is, a straight line.

  The voting unit 304 performs Hough voting on frequency components that satisfy all of the following conditions. Under this condition, only frequency components having a power equal to or higher than a predetermined threshold in a predetermined frequency band are voted.

(Voting condition 1) The frequency is in a predetermined range (low frequency cut and high frequency cut).

(Voting condition 2) The representative power P (fk) of the frequency component fk is greater than or equal to a predetermined threshold.

  The voting condition 1 is generally used for the purpose of cutting a low range where background noise is riding or cutting a high range where the FFT accuracy is lowered. The range of the low frequency cut and the high frequency cut can be adjusted according to the operation. When using the widest frequency band, it is appropriate to set the low frequency cut to only the DC component and the high frequency cut to the maximum frequency only.

  It is considered that the reliability of the FFT result is not high with a very weak frequency component such as background noise. The voting condition 2 is used for the purpose of preventing such frequency components having low reliability from participating in voting by thresholding with power. Assuming that the power value Po1 (fk) at the microphone 1a and the power value Po2 (fk) at the microphone 1b, the representative power P (fk) of the frequency component fk evaluated at this time is obtained as the average of both.

Further, the voting unit 304 adds a function value of the representative power P (fk) of the frequency component fk to the passing position of the trajectory at the time of voting. This voting method is a method that can obtain a higher maximum value if it contains frequency components with high power even if there are few points to pass, and has powerful components with high power even if there are few frequency components. Suitable for detecting straight lines (ie sound sources). The function value of the representative power P (fk) is calculated as G (P (fk)). FIG. 6 shows a calculation formula for G (P (fk)). The value of the intermediate parameter V is calculated as a value obtained by adding a predetermined offset α to the logarithmic value log ₁₀ (P (fk)) of P (fk). When V is positive, the value of V + 1 is set as the value of the function G (P (fk)). By voting at least 1 in this way, not only a straight line (sound source) containing a high-power frequency component rises to the top, but a straight line (sound source) containing many frequency components also rises to the top. It can have the same properties.

(Poll together multiple FFT results)
Further, the voting unit 304 can vote for each FFT, but generally, the voting unit 304 collectively votes for m consecutive (m ≧ 1) time-series FFT results. In the long term, the frequency components of the sound source will fluctuate, but by doing so, using more data obtained from the FFT results of moderately short time multiple times where the frequency components are stable You will be able to get a more reliable Hough voting result. Note that m can be set as a parameter according to the operation.

(Linear detection unit 305)
The straight line detection unit 305 is a means for detecting a powerful straight line by analyzing the vote distribution in the Hough voting space generated by the voting unit 304. At this time, by taking into consideration the circumstances peculiar to this problem such as the phase difference circulation described in FIG.

(Restriction of ρ = 0)
When the signals of the microphone 1a and the microphone 1b are A / D converted in phase by the acoustic signal input unit 2, the straight line to be detected always passes through ρ = 0, that is, the origin of the XY coordinate system. Therefore, the sound source estimation problem should ideally result in a problem of searching for a local maximum value from the vote distribution S (θ, 0) on the θ axis where ρ = 0 in the Hough voting space.

(Definition of straight line group considering phase difference circulation)
However, in reality, due to the cyclic nature of the phase difference, the straight line passing through the origin is translated by Δρ and circulated from the opposite side on the X axis is also a straight line showing the same arrival time difference. In this way, the straight line that passes through the origin and extends out of the X value range cyclically appears from the opposite side as the “circulation extension line”, and the straight line that passes through the reference origin as the “reference straight line”. I will call it. If the reference straight line is further inclined, the circulation extension line is further increased in number. If the coefficient a is an integer greater than or equal to 0, all straight lines having the same arrival time difference are a straight line group (θ0, aΔρ) obtained by translating the reference straight line defined by (θ0, 0) by Δρ. At this time, Δρ is a signed value defined by the equation shown in FIG. 7 as a function Δρ (θ) of the slope θ of the straight line.

(Maximum position detection considering phase difference circulation)
From the circulation of the phase difference, it was stated that the straight line representing the sound source should be treated as a straight line group consisting of a reference straight line and a circulation extension line instead of one. This must be taken into account when detecting the maximum position from the vote distribution.

  FIG. 8 shows the power spectrum of the frequency component for five times when processing is performed using actual speech spoken simultaneously from about 20 degrees left and about 45 degrees right in front of the microphone pair in a room noise environment. A phase difference scatter diagram for each frequency component obtained from the FFT result of m = 5) and a Hough vote result (voting distribution) obtained from the same five FFT results are shown.

  The amplitude data acquired by the microphone pair is converted into data of a power value and a phase value for each frequency component by the frequency resolving unit 301. In the figure, reference numerals 210 and 211 indicate logarithmic power values for each frequency component in luminance (the larger the black), the frequency is on the vertical axis and the time is on the horizontal axis. FIG. 6 is a diagram in which one vertical line corresponds to one frame (one FFT result) and is graphed over time (rightward). The upper stage 210 is the result of processing the signal from the microphone 1a and the lower stage 211 is the signal from the microphone 1b, and a large number of frequency components are detected. In response to this frequency decomposition result, the phase difference calculation unit 302 obtains the phase difference for each frequency component, and the scatter diagram generation unit 303 calculates the (x, y) coordinate value. 212 in the figure is a scatter diagram in which phase differences obtained by FFT for five consecutive frames from a certain time 213 are plotted. In this figure, a point cloud distribution along the reference line 214 inclined to the left from the origin and a point cloud distribution along the reference line 215 inclined to the right are recognized. Each point indicating such distribution is voted by the voting unit 304 to the Hough voting space to form a vote distribution 216.

  FIG. 9 is a diagram showing a result of searching for the maximum position by summing up several vote values separated by Δρ in consideration of phase difference circulation. The vote distribution 240 shown in FIG. 9A is obtained by displaying the positions of ρ by broken lines 242 to 249 when the straight line passing through the origin is translated by Δρ on the vote distribution 216 in FIG. At this time, the θ axis 241 and the broken lines 242 to 245 and the θ axis 241 and the broken lines 246 to 249 are spaced apart at equal intervals by a natural number multiple of Δρ (θ). It should be noted that no broken line is displayed at θ = 0 where it is certain that the straight line passes through the ceiling of the scatter diagram without exceeding the X value range.

  A given vote H (θ0) of θ0 is a total value of the votes on the θ-axis 241 and the votes on the broken lines 242-249 when viewed vertically at the position θ = θ0, that is, a is an integer of 0 or more. Calculated as H (θ0) = Σ {S (θ0, aΔρ (θ0))}. This operation is equivalent to adding up the votes of the reference straight line where θ = θ0 and the circulation extension line. A bar graph of this vote distribution H (θ) is 250 in FIG. From this vote distribution 250, ten local maximum positions indicated by reference numeral 251 in FIG. 9B are detected. Among these, the maximum positions 252 and 253 are straight line groups in which the sound from the left of the front of the microphone pair is detected about 20 degrees (corresponding to the maximum position 253, the reference straight line 254 and the circulation extension line 255 shown in FIG. 9C). Corresponding to the straight line group (the reference straight line 256 and the circulation extension lines 257 and 258 shown in FIG. 9C corresponding to the maximum position 252) in which the sound from the right of the microphone pair about 45 degrees right is detected. . In this way, by searching for the maximum position by summing the vote values at locations separated by Δρ, stable detection is possible from a straight line having a small inclination to a straight line having a large inclination. A candidate (sound source candidate) that seems to be a sound source can be extracted by selecting the maximum position (straight line) from which the vote value equal to or greater than the predetermined threshold is obtained.

(Presence angle estimation)
Furthermore, the straight line detection unit 305 calculates the existence range of sound source candidates corresponding to each straight line group from the detected θ value for each straight line group. When the distance to the sound source is sufficiently far from the distance between the microphones, the sound source exists within a certain angle (existence angle) with respect to a line segment (referred to as a base line of the microphone pair) connecting the two microphones 1a and 1b. It has a conical surface.

The arrival time difference ΔT between the microphone 1a and the microphone 1b can vary within a range of ± ΔTmax. When incident from the front of the microphone pair, ΔT is 0, and the sound source existence angle φ is 0 ° when the front is used as a reference. In addition, when the sound is incident directly to the right of the microphone pair, that is, from the direction of the microphone 1b, ΔT is equal to + ΔTmax, and the sound source existing angle φ is + 90 ° with the clockwise direction as a reference with respect to the front. Similarly, when sound enters from the left side of the microphone pair, that is, from the direction of the microphone 1a, ΔT is equal to −ΔTmax, and the existence angle φ is −90 °. In this way, ΔT is defined to be positive when sound enters from the right and negative when sound enters from the left. Considering general conditions based on the above, the existence angle including the sign can be calculated as φ = sin ⁻¹ (ΔT / ΔTmax).

  ΔTmax is a value obtained by dividing the inter-microphone distance L [m] by the sound velocity Vs [m / sec], which is obtained by ΔTmax = L ÷ Vs [sec]. At this time, it is known that the sound velocity Vs can be approximated by Vs = 331.4 + 0.604 t [m / sec] as a function of the temperature t [° C.]. Now, it is assumed that a straight line having an inclination θ is detected by the straight line detection unit 305. If this straight line is tilted to the right, θ is a negative value. When y = k (frequency fk), the phase difference ΔPh indicated by this straight line can be obtained as ΔPh (θ, k) = k · tan (−θ) as a function of k and θ. At this time, as shown by ΔT = (ΔPh (θ, k) / 2π) × (1 / fk), ΔT [sec] is equal to one period (1) of the frequency fk in proportion to 2π of the phase difference ΔPh (θ, k). / Fk) times multiplied by [sec]. Since θ is a signed quantity, ΔT is also a signed quantity. That is, when sound enters from the right (the phase difference ΔPh becomes a positive value), θ becomes a negative value. When sound enters from the left (the phase difference ΔPh becomes a negative value), θ becomes a positive value. Therefore, the sign of θ is inverted in the equation. In the actual calculation, the calculation may be performed with k = 1 (frequency immediately above the DC component k = 0).

(Time series tracking unit 306)
As described above, a straight line group is obtained by the straight line detection unit 305 for each Hough vote by the voting unit 304. Hough voting is performed on m consecutive (m ≧ 1) FFT results collectively. As a result, the straight line group is obtained in time series with the time corresponding to m frames as a period (hereinafter referred to as a “straight line detection period”). In addition, since θ of the straight line group has a one-to-one correspondence with the opening angle (existing angle) φ of the conical surface, it corresponds to a stable sound source regardless of whether the sound source is stationary or moving. It is assumed that the locus of (or φ) on the time axis changes continuously. On the other hand, the straight line group detected by the straight line detection unit 305 includes a straight line group corresponding to background noise (this will be referred to as a “noise straight line group”) depending on how the threshold is set. is there. However, it can be expected that the locus on the time axis of θ (or φ) of such a noise straight line group is not continuous or short even if it is continuous.

  The time-series tracking unit 306 collects θ (or φ) obtained for each straight line detection period into a group that can be regarded as continuous on the time axis in this way, so that the locus (sound source stream) of θ (or φ) on the time axis is collected. (Referred to as a candidate). With reference to FIG. 10, a grouping method when θ is used will be described.

  (1) A sound source stream candidate buffer is prepared. The sound source stream candidate buffer is an array of sound source stream candidate data. One sound source stream candidate data Kd can hold a start time Ts, an end time Te, an array of straight line group data Ld (straight line group list) constituting the sound source stream candidate, and a label number Ln. . One line group data Ld includes a θ value and a ρ value (by the line detection unit 305) of one line group constituting the sound source stream candidate, and a sound source existing angle φ value (line detection unit) corresponding to the line group. 305), a straight line score (by the straight line detection unit 305), and a time when they are acquired. As described above, because of the cyclic nature of the phase difference, the straight line representing the sound source should be treated as a straight line group consisting of a reference straight line and a cyclic extension line instead of one, so that 1 in the array of straight line group data Ld. One element is assumed to be one straight line group. If the circularity of the phase difference is not taken into account, the elements of the array are one straight line. The sound source stream candidate buffer is initially empty. A new label number is prepared as a parameter for issuing a label number, and an initial value is set to zero.

  (2) In a certain frame T, each newly detected straight line inclination θ (hereinafter referred to as θn and two indicated by black circle 323 and black circle 324 in the figure) is obtained in the sound source stream candidate buffer. Reference is made to the straight line group data Ld (black circles arranged in the rectangle in the figure) of the stored sound source stream candidate data Kd (rectangles 321 and 322 in the figure), and the difference between the θ value and θn (in the figure) 325 and 326) are within a predetermined angle threshold Δθ (giving an allowable range of angular gaps), and the difference between the acquisition times (327 and 328 in the figure) is a predetermined time threshold Δt (the allowable range of time direction gaps). The sound source stream candidate data having Ld within (given) is detected. As a result, the sound source stream candidate data 321 is detected for the black circle 323, but the nearest sound source stream candidate data 322 for the black circle 324 does not satisfy the above condition.

  (3) If sound source stream candidate data satisfying the condition (2) is found as indicated by the black circle 323, θn is assumed to form the same sound source stream candidate as this sound source stream candidate, and this θn and the corresponding The φ value, the ρ value, and the current time T are added to the line group list as new line group data of the sound source stream candidate data Kd, and the current time T is set as a new end time Te of the sound source stream candidate data. At this time, if a plurality of sound source stream candidate data are found, all of them are considered to form the same sound source stream candidate, and are integrated into the sound source stream candidate data having the youngest label number, and the rest are sound source stream candidate buffers. Delete from. The start time Ts of the integrated sound source stream candidate data is the earliest start time among the sound source stream candidate data before integration, and the end time Te is the latest end time of the sound source stream candidate data before integration. Yes, the line group list is a union of the line group lists of the sound source stream candidate data before integration. As a result, the black circle 323 is added to the sound source stream candidate data 321.

  (4) If no sound source stream candidate data satisfying the condition (2) is found, as indicated by a black circle 324, a new sound source stream candidate is set as the start of a new sound source stream candidate and a new sound source stream candidate in an empty portion of the sound source stream candidate buffer. The data is created, the start time Ts and the end time Te are both set as the current time T, θn, the corresponding φ value, ρ value, and the current time T are set as the first line group data in the line group list, and the new label number A value is given as the label number Ln of the sound source stream candidate data, and the new label number is incremented by one. When the new label number reaches a predetermined maximum value, the new label number is returned to zero. As a result, the black circle 324 is registered in the sound source stream candidate buffer as new sound source stream candidate data.

  (5) If there is sound source stream candidate data held in the sound source stream candidate buffer, the predetermined time Δt has elapsed from the last update (that is, from the end time Te) to the current time T. The sound source stream candidate data is output to the duration evaluation unit 307 in the next stage as a sound source stream candidate for which a new θn to be added has not been found, that is, grouping has been completed. In the example of the figure, the sound source stream candidate data 322 corresponds to this.

(Duration Evaluation Unit 307)
The duration evaluation unit 307 calculates the duration of the sound source stream candidate from the start time and end time of the sound source stream candidate data that has been grouped and output from the time series tracking unit 306, and sets the duration to a predetermined threshold. Those exceeding are recognized as (stable) sound source stream candidates based on sound source sounds, and others are recognized as (stable) sound source stream candidates based on noise. The sound source stream candidate data based on the sound source sound will be referred to as sound source stream information. The sound source stream information includes start time Ts and end time Te of the sound source stream, and time series data of θ, ρ, φ, and a straight line score indicating the sound source direction.

  Note that the number of straight line groups by the straight line detection unit 305 gives the number of candidates that seem to be sound sources, and includes noise sources. On the other hand, the number of sound source stream information by the duration evaluation unit 307 is considered to give the number of sound sources that can be trusted except for those that are supposed to be based on noise.

(Frame classification unit 308)
In the example of FIG. 10, the black circle 323 is determined to be data representing a series of sounds emitted from the same sound source as the sound source stream candidate data 321. At this time, the end of the sound source stream candidate data 321 is determined. Between the black circles 323 and the black circles 323, there was a gap period in which no straight line was detected. Consider this gap period as the dominant period of noise and assume that the separated speech during that period will not be at a reliable level. The frame classification unit 308 performs this determination.

  The frame classifying unit 308 gives a flag indicating whether or not reliability is possible to each frame of the sound source stream information output from the duration evaluation unit 307 using one of the following two reliability determination methods. Note that which method is used can be set according to the operation.

(Reliability determination method 1)
As illustrated in FIG. 11, a discrete time with the time axis cut at equal intervals is defined as a frame (331 in the figure). At this time, the sound source stream 332 includes a frame in which a straight line belonging to the sound source stream 332 can be detected (a time with a black circle in the figure) and a frame in which the straight line cannot be detected (333, 334, 335 in the figure). A straight line score (graphs 346, 347, and 348 in the figure) is given to a frame in which a straight line can be detected. In the reliability determination method 1, for each sound source stream information, a reliable flag is given to a frame in which a straight line has been detected, and an unreliable flag is given to a frame that is not.

(Reliability determination method 2)
As illustrated in FIG. 12, there is the same sound source stream 332 as in FIG. The vote distribution H (θ) is obtained for each frame by the Hough voting performed at each straight line detection cycle. In the figure, the vote distributions in four frames (time) are schematically shown as 339 to 342 in the figure. In the reliability determination method 2, the straight line inclination θ of a frame (333, 334, 335 in the figure) in which a straight line cannot be detected is calculated from frames before and after the straight line can be detected, assuming θ time continuity, for example. Estimate by interpolation with linear interpolation. The estimated θ is indicated by white circles 336, 337, and 338 in the figure. Then, the vote distribution H (θ) at the time corresponding to the θ value obtained by this interpolation is read. At this time, if the inclination (obtained by interpolation) of the frame 333 in which a straight line cannot be detected is θe (white circle 336 in the figure), the vote distribution H (θ) at that time (340 in the figure) The vote value H (θe) (343 in the figure) is read out and used as the straight line score of the frame 333. In this way, a straight line score of the gap period is obtained, and a straight line score (344 in the figure) over the entire stream is obtained together with the straight line score of the frame in which the straight line can be detected. Then, a reliable flag is given to a frame that has obtained a linear score equal to or higher than a predetermined threshold (345 in the figure), and an unreliable flag is given to other frames. By doing in this way, the reliability information can be generated by a threshold different from the threshold at the time of straight line detection.

(Sound source separation unit 4)
The internal configuration of the sound source separation unit 4 is shown in FIG. The sound source separation unit 4 includes an in-phase unit 371 and a beam forming unit 372.

(In-phase unit 371)
The in-phase unit 371 obtains the time transition of the sound source direction (existing angle) φ of the stream by referring to the sound source stream existing angle data, and obtains the intermediate value φmid = (from the maximum value φmax and the minimum value φmin of φ. φmax + φmin) / 2 is calculated to obtain the width φw = φmax−φmid. Then, the time Te of the time-series data of the two frequency-resolved data a and b that is the source of the sound source stream information is a time Te that has passed a predetermined time from the end time Te from a time Ts ′ that is a predetermined time before the start time Ts of the stream. It is roughly in-phased by extracting up to 'and correcting it so as to cancel the arrival time difference calculated backward with the intermediate value φmid.

(Beam forming part 372)
The time-series data of the two frequency decomposition data a and b roughly in-phased by the in-phase unit 371 is a signal as if it were incident from the front direction of the microphone pair. However, at each time, the front is not exactly 0 °, but changes within a range of ± φw. The beam forming unit 372 converts the rough and in-phase time-series data of the frequency-resolved data a and b into a margin of ± β · φw with respect to the front 0 ° (β is an appropriate coefficient of 1 or more, for example, 1.1), the sound source separation of the audio data of the stream is performed with high accuracy by applying to the “speaker tracking adaptive array” whose tracking range is the range of the angle given in 1.1). Emperor Amada et al., “Microphone array technology for speech recognition”, Toshiba Review 2004, VOL. 59, NO. 9, 2004, a configuration example of a speaker tracking adaptive array is disclosed.

(Vocabulary recognition unit 5)
The internal structure of the vocabulary recognition unit 5 is shown in FIG. The acoustic feature extraction unit 261 extracts time-series data of predetermined acoustic features (for example, MFCC, ΔMFCC, etc.) from the separated sound of the input sound source stream. This is called an input acoustic feature. The acoustic feature collating unit 262 collates the acoustic model stored in the acoustic model database 263 with the input acoustic feature, and converts the input acoustic feature string into predetermined phoneme symbol string (input series) data. The maximum likelihood sentence hypothesis search unit 264 generates an HMM (Hidden Markov Model) describing a sentence hypothesis from the grammar information stored in the grammar / language model database 265 and the language model, and the input sequence has the highest probability (likelihood). The sentence hypothesis to be output in (1) is searched for on the HMM and output. The output sentence hypothesis (maximum likelihood sentence hypothesis) is the result of recognizing the linguistic content of the sound source stream separated speech.

(Acoustic feature matching unit 362)
The acoustic model stored in the acoustic model database 363 is information obtained by combining standard acoustic features and their phonemic symbols for each phoneme. In addition, what phoneme should be used in Japanese is shown as a syllable table on page 45 of Kiyohiro Shikano et al., “Speech Recognition System”, Ohm Publishing Co., Ltd., published on May 15, 2001. .

  The acoustic feature matching unit 362 compares the input acoustic feature generated from the separated speech data with the standard acoustic feature described in the acoustic model, and calculates the similarity (acoustic score). Then, the phoneme symbols of the standard acoustic features up to the most similar top N (N best) and their acoustic scores are output. As a result, the input acoustic feature string is converted into a phoneme symbol string. However, the input acoustic feature sequence includes data of reliable frames and unreliable frames. At this time, since the noise of the unreliable frame data is more dominant than the target speech, it is unlikely that the input acoustic feature in that period is associated with an appropriate phoneme symbol. Therefore, the acoustic feature matching unit 362 associates a predetermined dummy phoneme symbol with a dummy acoustic score instead of matching the input acoustic feature of the unreliable frame with the standard acoustic feature. As a result, the verification process in the unreliable frame is omitted and the calculation cost is saved. Therefore, what is output as input state series data from the acoustic feature matching unit 362 is phoneme symbol string data including dummy phoneme symbols.

(Maximum likelihood sentence hypothesis search unit 364)
The sentence is composed of one or more words, and the grammatical information defines words assumed to appear in the utterance and their connection relations. Each word is composed of one or more phonemes, so that each word and sentence can be regarded as a phoneme string in which one or more phonemes are connected.

  An HMM is composed of states and transitions between states (including transitions to the same state). The state has a probability of being able to take that state (output probability), and the transition is a probability that the transition can occur (transition probability). To model the stochastic process. At this time, if the state is associated with a phoneme, a word that is a phoneme string can be described by an HMM (word HMM) in which a predetermined state is linked. A sentence can be described in a larger HMM in which predetermined word HMMs are chained. At this time, from a large number of example sentences (corpus), which phoneme follows each phoneme (biphone, triphone) and which word follows each word (bigram, trigram) is expressed by probability. The transition probability of the HMM can be set using a language model.

  When the initial state is symbolic sentence silence (SilB) and the word strings that make up a possible sentence are expanded into a tree structure, this is also placed at the end of the tree branch and leaves, which also reaches symbolic sentence silence (SilE). A phoneme string determined by the route to the final state represents a possible sentence (sentence hypothesis) defined by grammatical information.

  The separated speech is converted into a phoneme symbol string as an input sequence. The maximum likelihood sentence hypothesis searching unit 364 searches which sentence hypothesis on the HMM generated from the grammatical information can best explain this input sequence. A beam search is used for this search. Beam search is a technique in which several hypotheses that are promising at that time are left and the search is advanced while the remaining hypotheses that are not very promising are discarded. The criterion for how many hypotheses are left is called “beam width”, and the hypothesis discarding operation is called “pruning”.

  A state corresponding to the same phoneme symbol as the first phoneme symbol of the input phoneme symbol string (there is N at the maximum because N is the best) is searched for in all states that can transition from the initial state on the HMM. When a transitionable state corresponding to the same phoneme symbol is found, the output probability is the acoustic score of the phoneme, the transition probability from the initial state to the state is the language score, and the product of the acoustic score and the language score is Likelihood. Then, other routes are discarded (pruned) with the likelihood remaining up to the top M.

  As for the next phoneme symbol of the input phoneme symbol string (there is N at best, there are N at most), the state corresponding to the same phoneme symbol is searched in all the states that can be transitioned from the state remaining after pruning. . When such a state is found, an acoustic score and a language score are determined in the same manner, and the product is multiplied by the previous likelihood of the transition path leading to the new score. Similarly, the path is pruned with the new likelihood remaining up to the top M.

  The above processing is repeated until the end is reached, the transition path having the highest likelihood at the end arrival time is obtained, and the word string followed by the transition path is used as the linguistic interpretation of the separated speech data, that is, the recognition result.

  If the probabilities are treated logarithmically, all the products of probabilities and products of likelihood can be added. It is also possible to make the transition probabilities all 1 and make the likelihood depend only on the acoustic score. In addition, for post-processing after recognition, it is possible to output a transition path up to the top K position with a final likelihood as a recognition result (candidate).

  Although the recognition result is obtained as described above, a dummy phoneme symbol is included in the input series. The maximum likelihood sentence hypothesis search unit 364 processes the dummy phoneme symbols using one of the following two methods. It should be noted that which method is used can be changed by setting.

(Dummy support method 1: Stop pruning, stop likelihood calculation)
In this method, when the maximum likelihood sentence hypothesis search unit 364 encounters a dummy phoneme symbol, it leaves unpruned transitions from the currently valid state to all states that can be transited next. At this time, the likelihood of the remaining transition path is not updated. That is, only frames classified as reliable are used for likelihood calculation, and no pruning is performed on frames classified as unreliable (suppression of debranching). In this way, unlike silence and pauses that are not originally uttered, even if the target voice is covered with noise during the utterance, the recognition process can be continued while calculating the likelihood over the period. become able to. In this way, even if the unreliable period coincides with the silent section, it is different from Patent Document 2 that no pruning is performed during that period. This difference is not only the period in which the present invention attempts to detect as unreliable, but the period in which the target voice is actually silent, as well as the target voice is present, making it difficult to hear due to noise. This is because the period is also included. The next time the target voice begins to be heard, this is a response to the situation that the utterance has progressed far ahead.

(Dummy support method 2: Insert dummy phoneme state)
In this method, the maximum likelihood sentence hypothesis search unit 364 inserts a transition to a state corresponding to a dummy phoneme in parallel with all transition destinations on the HMM. When a dummy phoneme symbol is encountered, a transition from the currently valid state to the dummy phoneme state occurs. By setting the output probability (acoustic score) of the dummy phoneme state to 1 and also entering 1 in the transition probability (language score) there, the likelihood of the transition path is not changed. That is, all the subsequent calculations can be performed as usual only by generating an HMM that takes dummy phonemes into account, and exception processing as in the dummy handling method 1 is not required. However, since the scale of the HMM is expanded by inserting the dummy phoneme state, this method is preferably used for small grammatical information.

  Note that the acoustic feature matching unit 362 does not associate the dummy phoneme symbol with the unreliable frame, but the maximum likelihood sentence hypothesis searching unit 364 directly refers to the reliability information to perform the above-described exception processing (dummy handling method 1). It is also possible to do.

(Speaker recognition unit 6)
The speaker recognizing unit 6 recognizes who the voice of the input sound source stream is, for example, Mr. A's voice. The internal configuration of the speaker recognition unit 6 for this purpose is shown in FIG. The acoustic feature extraction unit 271 extracts time-series data of a predetermined acoustic feature (for example, formant) from only the sound of the reliable frame among the separated sounds of the input sound source stream. This is called an input acoustic feature for speaker recognition. The acoustic feature matching unit 272 compares the standard acoustic features for speaker recognition stored in the standard speaker feature database 273 for each speaker with the input acoustic features for speaker recognition, and performs speaker recognition. The average similarity over the entire input acoustic feature sequence is calculated for each speaker. The speaker whose average similarity is equal to or greater than a predetermined threshold is identified as the speaker of the separated speech, and the speaker ID is output. If the average similarity equal to or greater than the threshold is not obtained, a special ID indicating that the speaker of the separated speech is unknown is output.

(Sound recognition unit 7)
The sound object recognition unit 7 recognizes what kind of sound the separated sound of the input sound source stream is, for example, “It is a sound that breaks the glass”. FIG. 16 shows the internal configuration of the sound recognition unit 7 for that purpose. The acoustic feature extraction unit 281 has predetermined acoustic features (for example, an envelope, a logarithmic power spectrum, etc., for example, from the sound of a reliable frame among the separated sound of the input sound source stream, for example, if the sound that breaks the glass is seen in the time direction. Time-series data is extracted that shows an attenuating envelope with decreasing amplitude and whose logarithmic power spectrum is close to white. This is called an input acoustic feature for object sound recognition. The acoustic feature collating unit 282 collates the standard acoustic features for recognizing the sound for each sound stored in the standard sound feature database 283 with the input sound features for recognizing the sound, and the entire input sound feature sequence for the sound recognition. The average of the similarity over is calculated for each sound. The sound with the average similarity equal to or greater than a predetermined threshold is recognized as the identity of the separated sound, and the sound ID is output. If an average similarity equal to or greater than the threshold is not obtained, a special ID indicating that the identity of the separated speech is unknown is output.

(Output unit 8)
The output unit 8 includes the number of sound sources, the spatial existence range of each sound source (the existence angle φ that determines the conical surface), the temporal existence period (Ts, Te) of the sound emitted from the sound source, It is a means for outputting sound source information including at least one of separated speech, linguistic content of the separated speech, a speaker of the separated speech, and a physical sound of the separated speech.

(User interface unit 9)
The user interface unit 9 presents various setting contents necessary for the above-described voice recognition processing to the user, accepts setting input from the user, saves the setting contents in the external storage device, and reads out from the external storage device 8 and FIG. 9 (1) display of frequency components for each microphone, (2) display of scatter diagram, (3) display of various vote distributions, (4) display of maximum position, (5) Various processing results and intermediate results are visualized and displayed to the user, such as the display of the straight line group on the scatter diagram and the display of the (6) sound source stream candidate data shown in FIG. It is a means for making it select and visualizing in detail. In this way, the user can confirm the operation of the voice recognition apparatus according to the present embodiment, adjust the voice recognition apparatus to perform a desired operation, and thereafter use the apparatus in an adjusted state. It becomes possible to do.

(Process flow diagram)
FIG. 17 shows the flow of processing in the speech recognition apparatus according to this embodiment. This processing includes initial setting processing step S1, acoustic signal input processing step S2, sound source stream extraction / classification processing step S3, sound source separation processing step S4, vocabulary recognition processing step S5, speaker recognition processing step S6, It consists of a sound recognition processing step S7, an output processing step S8, an end determination processing step S9, a confirmation determination processing step S10, an information presentation / setting acceptance processing step S11, and an end processing step S12.

  The initial setting processing step S1 is a processing step for executing a part of the processing in the user interface unit 9, and reads various setting contents necessary for the voice recognition processing from the external storage device, and initializes the device to a predetermined setting state. To do.

  The acoustic signal input processing step S2 is a processing step for executing processing in the acoustic signal input unit 2, and inputs two acoustic signals captured at two positions that are not spatially identical.

  The sound source stream extraction / classification processing step S3 is a processing step for executing processing in the sound source stream extraction / classification unit 3, and (1) frequency-resolves each of the two input acoustic signals in the acoustic signal input processing step S2, and (2) The phase difference value for each frequency of both input acoustic signals is calculated, and the phase difference value for each frequency is generated as scatter diagram data on the XY coordinate system with the frequency as the Y axis and the phase difference value as the X axis. 3) A predetermined straight line is detected from the scatter diagram data. (4) Based on the detected straight line information, the number of sound sources that generate the acoustic signal, the spatial existence range of each sound source, and each sound source (5) Each time of the existence period is classified as reliable or unreliable, and these pieces of information are output as sound source stream information.

  The sound source separation processing step S4 is a processing step for executing processing in the sound source separation unit 4, and separates and extracts the sound of each sound source based on the sound source stream information.

  The vocabulary recognition processing step S5 is a processing step for executing the processing in the vocabulary recognition unit 5, and recognizes the linguistic meaning of the separated and extracted speech of each sound source.

  The speaker recognition processing step S6 is a processing step for executing the processing in the speaker recognition unit 6, and recognizes the speaker type of the separated and extracted speech of each sound source.

  The object sound recognition processing step S7 is a process step for executing the process in the object sound recognition unit 7, and recognizes what object sound the separated and extracted sound of each sound source is.

  The output processing step S8 is a processing step for executing the processing in the output unit 8, and outputs the sound source stream information and the result of the voice recognition.

  The end determination process step S9 is a process step for executing a part of the process in the user interface unit 9. The end determination process step S9 checks whether or not there is an end instruction from the user. If there is no left branch), the flow proceeds to the confirmation judgment processing step S10 (upper branch) and the process flow is controlled.

  The confirmation determination processing step S10 is a processing step for executing a part of the processing in the user interface unit 9. The presence / absence of a confirmation command from the user is inspected, and if there is a confirmation command, an information presentation / setting acceptance processing is performed. If not, go to step S11 (left branch), and if not, control the acoustic signal processing step S2 (upper branch) and the flow of processing.

  The information presentation / setting acceptance processing step S11 is a processing step for executing a part of the processing in the user interface unit 9 that is executed in response to a confirmation command from the user. Presenting to the user, accepting setting input from the user, saving the setting contents to the external storage device by the save command, reading the setting contents from the external storage device by the read command, various processing results and intermediate results So that the user can confirm the operation of the voice recognition processing or perform the desired operation by making the user select the desired data and visualize it in more detail It is possible to make adjustments and continue processing in an adjusted state thereafter.

  The termination processing step S12 is a processing step for executing a part of the processing in the user interface unit 9 that is executed in response to a termination command from the user. To the external storage device for various setting contents necessary for the speech recognition processing. Automatically saves.

  Hereinafter, some modifications of the above-described embodiment will be described.

(Multiple systems mounted in parallel)
The above example has been described with the simplest configuration including two microphones. However, as shown in FIG. 18, N (N ≧ 3) microphones are provided, and a maximum of M (1 ≦ M ≦ _N C ₂ It is also possible to configure a pair of microphones.

Reference numerals 11 to 13 in the figure denote N microphones. In the figure, reference numeral 22 denotes means for inputting N acoustic signals from N microphones. Reference numeral 23 in the figure shows frequency analysis of the input N acoustic signals, respectively, and scatter diagram data for each of a pair of M (1 ≦ M ≦ _N C ₂ ) pairs composed of two of the N acoustic signals. Means for detecting a predetermined straight line from the generated M sets of scatter diagram data and generating sound source stream information including frame reliability information from each of the detected information of the M sets of straight lines. is there. Reference numeral 24 in the figure denotes means for extracting separated sound of each sound source using the generated sound source stream information. Reference numeral 25 in the figure denotes means for recognizing the linguistic content of the extracted separated speech. In the figure, reference numeral 26 denotes means for recognizing different speakers of the extracted separated speech. In the figure, reference numeral 27 denotes means for recognizing different extracted sounds of separated speech. In the figure, 28 is a means for outputting sound source stream information and the result of speech recognition. 29 in the figure shows various setting values including information of microphones constituting each pair to the user, accepts setting inputs from the user, saves the setting values to the external storage device, and setting values from the external storage device Is a means for executing reading of data and presenting various processing results to the user. The processing in each microphone pair is the same as in the embodiments described so far, and such processing is executed in parallel for a plurality of microphone pairs.

  In this way, even if there are gains and weaknesses in the sound source direction for each microphone pair, it is possible to detect, localize and recognize target sound sources that exist in a wide range of directions around the device by covering with multiple microphone pairs It becomes possible to do.

(Implementation using a computer: program)
The present invention can also be implemented using a computer as shown in FIG. Reference numerals 31 to 33 in the figure denote N microphones. In the figure, 40 is an A / D conversion means for inputting N acoustic signals from N microphones, and 41 in the figure is a CPU for executing a program command for processing the inputted N acoustic signals. It is. 42 to 47 in the figure are standard devices constituting the computer, and are a RAM 42, a ROM 43, an HDD 44, a mouse / keyboard 45, a display 46, and a LAN 47, respectively. Reference numerals 50 to 52 in the figure denote drives for supplying programs and data to the computer from the outside via a storage medium, which are a CD ROM 50, an FDD 51, and a CF / SD card 52, respectively. Reference numeral 48 in the figure denotes D / A conversion means for outputting an acoustic signal, and a speaker 49 is connected to the output. This computer device functions as an acoustic signal processing device by storing an acoustic signal processing program comprising the processing steps shown in FIG. 27 in the HDD 44, reading it into the RAM 42 and executing it by the CPU 41. Further, the functions of the user interface unit 8 described above are realized by using the HDD 44 as an external storage device, the mouse / keyboard 45 that receives operation input, the display 46 and the speaker 49 as information presenting means. The sound source information obtained by the acoustic signal processing is stored and output to the RAM 42, ROM 43, and HDD 44, or communicated and output via the LAN 47.

(recoding media)
The present invention can also be implemented as a recording medium as shown in FIG. Reference numeral 61 in the figure denotes a recording medium realized by a CD-ROM, CF, SD card, floppy (registered trademark) disk or the like on which a signal processing program according to the present invention is recorded. The program can be executed by inserting the recording medium 61 into an electronic device 62 such as a television or a computer, an electronic device 63, or a robot 64, or another electronic device 65 is communicated from the electronic device 63 supplied with the program. By supplying the program to the robot 64, the program can be executed on the electronic device 65 or the robot 64.

  According to the embodiment of the present invention described above, the following operational effects can be obtained.

  (1) The sound source stream extraction and classification means can detect a frame in which noise is dominant among all the frames of the target sound source stream to be recognized. Then, whether or not noise is dominant is given as reliability information (reliable flag / unreliable flag) for each frame, so that the information can be used by various subsequent voice recognition means.

  As compared with Non-Patent Document 1, the conventional technique determines whether each element (frequency component) of the acoustic feature is reliable based on the noise estimated by the sound source separation means, while comparing with the present invention. Is for determining the reliability in the sound source detection process prior to the sound source separation process.

  (2) When converting the input acoustic feature sequence into a phoneme symbol sequence, the vocabulary recognition means immediately converts the acoustic feature of the unreliable frame into a dummy phoneme without matching with the standard acoustic feature, thereby calculating the acoustic feature comparison. Cost can be reduced.

  (3) When searching for a sentence hypothesis that matches the input acoustic feature sequence (input phoneme symbol sequence), the vocabulary recognition means allows only the reliable frame to participate in the likelihood calculation and does not prune the unreliable frame. Thus, it is possible to prevent the search from being broken and to suppress the occurrence of erroneous recognition in a noisy environment.

  (4) When converting the input acoustic feature sequence to the phoneme symbol sequence, the vocabulary recognition means immediately converts the acoustic feature of the unreliable frame into the dummy phoneme without matching the standard acoustic feature, and the input acoustic feature sequence When searching for sentence hypotheses that match (input phoneme symbol string), search hypotheses are evaluated using a search tree that takes into account dummy phonemes to prevent search failures and prevent misrecognition in noisy environments. can do.

  With regard to the above three points, in particular, compared with Patent Document 2, the prior art has detected the silent period and narrowed the beam width, but this is intended to reduce the calculation cost in the silent period. . This operation is valid during the pause period during which the silence period is speaking, that is, a part of the speech being spoken during that period is not lost, and the consistency with the sentence hypothesis being searched cannot be lost. Sometimes limited. On the other hand, the present invention is made on the assumption that the target speech that is defeated by noise is missing during the unreliable period. Therefore, in order to prevent the failure of the search due to this omission, the reliability information is used, and an operation for expanding the beam width is performed during this period. As a result, even if there is a period in which the target speech is defeated by noise in a noisy environment, it realizes speech recognition that does not fail in that period and can continue to recognize the speech of the period that is not defeated Yes.

  (5) When recognizing the input speech feature with the standard speaker feature, the speaker recognizing means collates only the speech feature of the reliable frame with the standard speaker feature, thereby preventing erroneous recognition in a noisy environment. Can be suppressed.

  (6) When the input sound feature is compared with the standard sound feature, the sound recognition means checks only the sound feature of the reliable frame with the standard sound feature, thereby suppressing the occurrence of misrecognition in a noisy environment. Can do.

  With regard to the above two points, it is considered that the class of the speaker belongs to the class of the sound. Therefore, the standard acoustic feature and the input feature of each class are collated, and the class having the highest similarity is recognized as the speech class. This is true for recognition in general. At this time, if it tries to recognize bad data, it will make a mistake called misrecognition. If only good data can be selected and recognized, the results will be better. According to the present invention, since the reliability information gives information on whether the data is good or bad (speech clarity), only good data can be selected and evaluated in the recognition process.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

Functional block diagram of a speech recognition apparatus according to an embodiment of the present invention Block diagram showing the internal configuration of the sound source stream extraction and classification unit Explanation of phase difference calculation Illustration of coordinate value calculation Phase difference circulation diagram Illustration of function value of average power voted Relationship diagram between θ and Δρ A diagram showing frequency components, scatter diagrams, and Hough voting results during simultaneous speech A figure showing the result of searching for the maximum position by summing the votes obtained at several points separated by Δρ Diagram for explaining tracking of θ on the time axis The figure for demonstrating the reliability determination method 1 The figure for demonstrating the reliability determination method 2 The figure for demonstrating the internal structure of a sound source separation part Block diagram showing the internal structure of the vocabulary recognition unit Block diagram showing the internal structure of the speaker recognition unit Block diagram showing the internal structure of the sound recognition unit The flowchart which showed the flow of the speech recognition process which concerns on one Embodiment of this invention. Functional block diagram showing a modified embodiment using N microphones Functional block diagram showing an embodiment using a computer The figure which shows embodiment by a recording medium

Explanation of symbols

DESCRIPTION OF SYMBOLS 1a, 1b ... Microphone, 2 ... Sound signal input part, 3 ... Sound source stream extraction classification | category part, 4 ... Sound source separation part, 5 ... Vocabulary recognition part, 6 ... Speaker recognition part, 7 ... Object recognition part, 8 ... Output part , 9 ... User interface part

Claims (6)

Input means for inputting the first and second acoustic signals captured at two points;
Calculating means for frequency-decomposing each of the first and second acoustic signals to obtain a frequency component, and calculating a phase difference and power for each frequency component;
Generating means for generating a scatter diagram having the value of the frequency component and the value of the phase difference as coordinate values;
Detection means for detecting an arrangement of frequency components showing linearity on the scatter diagram together with a linear score corresponding to the power, and detecting an arrangement of frequency components having the linear score equal to or greater than a threshold as a straight line indicating the presence of a sound source When,
A sound source stream that groups at least one straight line detected by the detecting means in a time axis direction while allowing a straight line non-detection period and straight line tilt fluctuation within a certain range, the information including the straight line inclination, Extraction means for extracting a sound source stream including a straight line score and information of a time when the straight line is detected;
Classifying means for assigning reliability information based on the level of the straight line score to the time of the sound source stream, and classifying each frame of the sound source stream;
Sound source separation means for extracting sound data of the sound source stream based on the sound source existence angle calculated from the information of the slope of the straight line included in the sound source stream, and separating the sound source;
The sentence hypothesis defined in the grammatical information is expanded into a state and transition search tree, a predetermined acoustic feature is extracted from the sound data of the sound source stream, and the likelihood of the state transition path of the search tree for the acoustic feature sequence And voice recognition means for recognizing the linguistic content of the sound source stream by searching for a state transition path with a high likelihood,
A speech recognition apparatus that controls search of the state transition route based on the reliability information. The speech recognition means performs a beam search with pruning based on likelihood in the search,
Calculate the likelihood for times classified as trustworthy,
The speech recognition apparatus according to claim 1, wherein the pruning is suppressed for the time classified as unreliable. The speech recognition means uses a search tree in which a dummy state is added in parallel with each state of the search tree in the search,
For times classified as trustworthy, transition to a state other than the dummy state,
The speech recognition apparatus according to claim 1, wherein a transition to the dummy state is performed for the time classified as unreliable. Input means for inputting the first and second acoustic signals captured at two points;
Calculating means for frequency-decomposing each of the first and second acoustic signals to obtain a frequency component, and calculating a phase difference and power for each frequency component;
Generating means for generating a scatter diagram having the value of the frequency component and the value of the phase difference as coordinate values;
Detection means for detecting an arrangement of frequency components showing linearity on the scatter diagram together with a linear score corresponding to the power, and detecting an arrangement of frequency components having the linear score equal to or greater than a threshold as a straight line indicating the presence of a sound source When,
A sound source stream that groups at least one straight line detected by the detecting means in a time axis direction while allowing a straight line non-detection period and straight line tilt fluctuation within a certain range, the information including the straight line inclination, A sound source stream extracting means for extracting a sound source stream including a straight line score and information of a time when the straight line is detected;
Classifying means for assigning reliability information based on the level of the straight line score to the time of the sound source stream, and classifying each frame of the sound source stream;
Sound source separation means for extracting sound data of the sound source stream based on the sound source existence angle calculated from the information of the slope of the straight line included in the sound source stream, and separating the sound source;
Feature extraction means for extracting a predetermined feature from audio data at a time determined to be reliable by the reliability information among audio data of the sound source stream;
A calculation means for calculating a similarity between the feature and a class feature learned in advance for each class to be identified;
Recognizing means for recognizing the class of the class feature having the highest similarity as the class of the voice data. An input step for inputting first and second acoustic signals captured at two points;
A calculation step of frequency-decomposing each of the first and second acoustic signals to obtain a frequency component, and calculating a phase difference and power for each frequency component;
A generation step of generating a scatter diagram having the value of the frequency component and the value of the phase difference as coordinate values;
A detection step of detecting an arrangement of frequency components showing linearity on the scatter diagram together with a linear score corresponding to the power, and detecting an arrangement of frequency components having the linear score equal to or greater than a threshold as a straight line indicating the presence of a sound source. When,
A sound source stream that groups at least one straight line detected by the detection step in a time axis direction while allowing a straight line non-detection period and straight line tilt fluctuation within a certain range, and includes information including the slope of the straight line, An extraction step of extracting a sound source stream including a straight line score and information of a time when the straight line is detected;
A classification step of assigning reliability information based on the level of the straight line score to the time of the sound source stream, and classifying each frame of the sound source stream;
A sound source separation step of extracting sound data of the sound source stream based on a sound source existing angle calculated from information on the slope of the straight line included in the sound source stream, and separating the sound source;
The sentence hypothesis defined in the grammatical information is expanded into a state and transition search tree, a predetermined acoustic feature is extracted from the sound data of the sound source stream, and the likelihood of the state transition path of the search tree for the acoustic feature sequence And recognizing the linguistic content of the sound source stream by searching for a highly likely state transition path,
A speech recognition method, wherein search for the state transition route is controlled based on the reliability information. An input step for inputting first and second acoustic signals captured at two points;
A calculation step of frequency-decomposing each of the first and second acoustic signals to obtain a frequency component, and calculating a phase difference and power for each frequency component;
A generation step of generating a scatter diagram having the value of the frequency component and the value of the phase difference as coordinate values;
A detection step of detecting an arrangement of frequency components showing linearity on the scatter diagram together with a linear score corresponding to the power, and detecting an arrangement of frequency components having the linear score equal to or greater than a threshold as a straight line indicating the presence of a sound source. When,
A sound source stream that groups at least one straight line detected by the detection step in a time axis direction while allowing a straight line non-detection period and straight line tilt fluctuation within a certain range, and includes information including the slope of the straight line, A sound source stream extracting step of extracting a sound source stream including a straight line score and information of a time when the straight line is detected;
A classification step of assigning reliability information based on the level of the straight line score to the time of the sound source stream, and classifying each frame of the sound source stream;
A sound source separation step of extracting sound data of the sound source stream based on a sound source existing angle calculated from information on the slope of the straight line included in the sound source stream, and separating the sound source;
A feature extraction step of extracting a predetermined feature from audio data at a time determined to be reliable by the reliability information among the audio data of the sound source stream;
A calculation step for calculating a similarity between the feature and a class feature learned in advance for each class to be identified;
And a recognition step for recognizing the class of the class feature having the highest similarity as the class of the voice data.

Images