JP2018081239A - Voice processing method, voice processing device, and voice processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To set the utterance start position in voice data correctly.SOLUTION: The voice processing method includes a computer exerting the following processing. When plural microphones detect the utterance from the plural voice data which are collected at the same time according to the prescribed condition, the computer firstly sets the utterance start position, based on the prescribed condition, back to the past from the position of the detection of the utterance in the voice data. Next, the computer identifies the direction of the speaker viewed from the plural microphones, based on the characteristic difference from the utterance start position to the position of the detection of the utterance in each of the plural voice data. Next, the computer, based on the direction of the identified speaker, extracts the utterance section after the utterance start position in either one of the plural voice data.SELECTED DRAWING: Figure 6A

Description

  The present invention relates to a voice processing method, a voice processing apparatus, and a voice processing program.

  As one of the technologies for translating the utterance content of voice data, a speaker is identified based on a plurality of voice data picked up simultaneously by a plurality of microphones, and a source language and a destination language are determined according to the result of the speaker's identification. There is a technology to translate voice data by switching between and. In this type of technology, for example, after specifying a speaker based on a characteristic difference between utterance sections in a plurality of voice data, the user selects any one of the plurality of voice data based on the result of the speaker's specification, and utterance contents Is translated (see, for example, Patent Document 1). Further, in this type of technology, for example, according to the direction of a sound source (speaker) estimated from a plurality of sound data, processing for separating sound sources included in sound data and processing for translation and display are controlled ( For example, see Patent Document 2).

  Furthermore, when detecting an utterance from voice data, the voice data is divided into a plurality of frames, and it is determined whether or not a significant voice is included in each frame by Voice Activity Ditection (VAD) or the like. Then, when a predetermined number of frames including significant speech continues, speech detection is confirmed.

JP 2005-148301 A JP2015-106014A

  In the method of determining the detection of an utterance when a predetermined number of frames including significant speech are continued, the detection of the utterance is determined at a position delayed by several frames from the frame where the utterance is actually started. For this reason, for example, when the utterance content in the utterance section is translated using the frame in which the detection of the utterance is confirmed as the start position of the utterance section, the beginning of the word is cut off.

  In addition, when a frame preceding a predetermined fixed length from the frame in which the detection of the utterance is determined in consideration of the above delay is set as the start position of the utterance section, a silent frame before the frame in which the utterance actually starts And noise frames may be included in the speech interval.

  In one aspect, an object of the present invention is to correctly set an utterance start position in audio data.

  In the voice processing method according to one aspect, the computer executes the following processing. First, when a computer detects an utterance according to a predetermined condition from a plurality of audio data collected simultaneously by a plurality of microphones, the computer goes back to the past from the position where the utterance is detected in the audio data based on the predetermined condition. Set the start position of the utterance. Next, the computer identifies the direction of the speaker viewed from the plurality of microphones based on the characteristic difference from the start position of the utterance to the position where the utterance is detected in each of the plurality of audio data. Next, the computer extracts an utterance section after the utterance start position in any of the plurality of audio data based on the direction of the identified speaker.

  According to the above aspect, it is possible to correctly set the utterance start position in the audio data.

It is a block diagram which shows the functional structure of the speech processing unit which concerns on 1st Embodiment. It is a block diagram which shows the structural example of a direction identification part. It is a figure which shows the example of the positional relationship of a microphone and a speaker. It is a figure which shows the example of a setting of speaker information. It is a flowchart explaining the translation process of the utterance content which concerns on 1st Embodiment. It is a flowchart (the 1) explaining the content of an utterance detection process. It is a flowchart (the 2) explaining the content of an utterance detection process. It is a flowchart explaining the content of a direction identification process. It is a figure explaining the process which a 1st audio | voice process part performs. It is a figure explaining the example of the determination method of whether a frame is sound. It is a figure which shows an example of the process result of an utterance detection process. It is a figure which shows another example of the process result of an utterance detection process. It is a figure explaining the calculation method of the average power in a direction identification process. It is a figure explaining the relationship between a sound pressure difference and the direction of a speaker. It is a figure explaining the switching method of a translation language. It is a flowchart explaining the translation process of the utterance content which concerns on 2nd Embodiment. It is a figure which shows the example of the translation process which can occur in the audio | voice process which concerns on 1st Embodiment. It is a figure which shows the example of the translation process by the audio | voice process which concerns on 2nd Embodiment. It is a block diagram which shows the functional structure of the audio processing apparatus and server apparatus which concern on 3rd Embodiment. It is a figure which shows the hardware constitutions of a computer.

<First Embodiment>
FIG. 1 is a block diagram showing a functional configuration of the speech processing apparatus according to the first embodiment.

  As shown in FIG. 1, the speech processing apparatus 1 of this embodiment includes a first speech processing unit 110, a second speech processing unit 120, and a speaker information setting unit 130. The speech processing apparatus 1 according to the present embodiment also includes a speech data holding unit 191, a calculation result holding unit 192, a speaker information holding unit 193, and a translation dictionary 194.

  The first audio processing unit 110 acquires audio data from each of a plurality of microphones 201 and 202 capable of separately outputting audio data collected simultaneously, and holds the audio data in the audio data holding unit 191 in units of frames. To perform the process. The first audio processing unit 110 includes an audio data acquisition unit 111 and a retained data control unit 112. In the following description, the microphone is simply referred to as “microphone”.

  The audio data acquisition unit 111 acquires audio data collected simultaneously from each of the plurality of microphones 201 and 202. The plurality of microphones 201 and 202 are, for example, a set of an L channel microphone 201 and an R channel microphone 202 in one stereo microphone 2.

  The holding data control unit 112 divides the acquired audio data into frames (processing units) for each predetermined time length, and causes the audio data holding unit 191 to hold each frame in an identifiable state. Also, the retained data control unit 112 deletes frames that are no longer needed from among the frames retained by the audio data retaining unit 191.

  The second speech processing unit 120 detects a speech segment in speech data acquired from each of the plurality of microphones 201 and 202, and identifies the direction in which a speaker (speaker) speaking in the speech segment exists. . At this time, the second voice processing unit 120 sets the start position of the utterance section by going back a predetermined time length (frame) from the position where the utterance is detected in the voice data. Further, the second audio processing unit 120 identifies whether the speaker in the utterance section exists in the left or right direction when viewed from the stereo microphone 2 based on the characteristic difference between the plurality of audio data. Furthermore, the second speech processing unit 120 in the speech processing device 1 according to the present embodiment changes the utterance content of the utterance section from the language associated with the utterer to another language based on the identification result of the utterer in the utterance section. Process to translate to. The second speech processing unit 120 in the speech processing apparatus 1 of the present embodiment includes an utterance detection unit 121, a direction identification unit 122, a language switching unit 123, a speech data extraction unit 124, a translation processing unit 125, and an output. Part 126.

  The utterance detection unit 121 determines whether or not there is sound for each frame of audio data, and detects the utterance section of each audio data based on the determination result. Here, a frame which is sound is a frame including a significant audio component such as a speaker's utterance. The utterance detection unit 121 determines whether the frame is sounded by a known determination method. Further, the utterance detection unit 121 counts the number of frames that are sound in a plurality of consecutive frames including the current processing target frame for each audio data. Then, when the number of sound frames in a plurality of consecutive frames is equal to or greater than the threshold, the utterance detection unit 121 detects an utterance section of audio data including the plurality of frames. At this time, the utterance detection unit 121 counts the number of frames that are uttered from the frame in which the utterance is detected to the past frames, and the utterance detection frame having the utterance detection delay length d_frame is the utterance interval. This is the start frame.

  The direction identifying unit 122 identifies the direction of the speaker based on the difference in characteristics between frames in the utterance section detected from each voice data. For example, the direction identification unit 122 is based on the sound pressure difference between the first audio data frame acquired from the L channel microphone 201 and the second audio data frame acquired from the R channel microphone 202. Identify the direction of the speaker. When audio data is acquired by the stereo microphone 2 including the L-channel microphone 201 and the R-channel microphone 202, the direction identification unit 122 moves in either the left direction or the right direction of the stereo microphone 2. Identify if the speaker is present.

  The translation language switching unit 123 switches the combination of the translation source language and the translation destination language based on the direction of the speaker identified by the direction identification unit 122 and preset speaker information. The speaker information includes the direction in which the speaker is seen from the stereo microphone 2 and the language in which the speaker is uttered. For example, the user of the speech processing apparatus 1 inputs the speaker information by performing a predetermined operation on the speaker information setting unit 130. The speaker information setting unit 130 causes the speaker information holding unit 193 to hold the speaker information input by the user. For example, it is assumed that the utterance language of the speaker in the direction of the microphone 201 for the L channel is Japanese, and the utterance language of the speaker in the direction of the microphone 202 for the R channel is English. In this case, if the direction of the identified speaker is the direction of the microphone 201 for the L channel, the translation language switching unit 123 sets the translation source language to Japanese and the translation destination language to English. Thereafter, when the direction of the identified speaker changes to the direction of the microphone 202 for the R channel, the translation language switching unit 123 sets the translation source language to English and the translation destination language to Japanese.

  Based on the direction of the speaker identified by the direction identifying unit 122, the speech data extracting unit 124 extracts a speech section of speech data used for translation processing from a plurality of speech data. For example, when the direction of the speaker is the direction of the L channel microphone 201, the audio data extraction unit 124 includes the audio data acquired from the L channel microphone 201 held by the audio data holding unit 191. Extract the utterance section to translate.

  The translation processing unit 125 analyzes the utterance content in the utterance section of the voice data extracted by the voice data extraction unit 124 and translates it from the source language into the target language. The translation processing unit 125 generates document data obtained by translating (converted) the utterance content of the utterance section from the translation source language to the translation destination language by a known translation method. For example, the translation processing unit 125 first extracts a character string indicating the utterance content from the frame data of the utterance section based on the voice characteristics and the like in the translation source language, and converts the extracted character string into a sentence. After that, the translation processing unit 125 translates (converts) the character string converted into a text in the translation source language into text data in the translation destination language. At this time, the translation processing unit 125 refers to the word dictionary for morphological analysis included in the translation dictionary 194 and converts the utterance contents into sentences. In addition, the translation processing unit 125 refers to the translation dictionary included in the translation dictionary 194 and translates the text string converted into text data in the translation destination language.

  The output unit 126 visualizes the text data in the translation destination language generated by the translation processing unit 125 and outputs the text data to the display device 3. Note that the output unit 126 may output the text data in the translation destination language generated by the translation processing unit 125 as audio data to a speaker (not shown).

  As described above, the audio data holding unit 191 holds the audio data acquired from the plurality of microphones 201 and 202 in units of frames. The audio data holding unit 191 holds a plurality of past frames including a frame that is a processing target in the processing in the second audio processing unit 120. The calculation result holding unit 192 holds the results of various calculation processes in the second sound processing unit 120. As described above, the speaker information holding unit 193 holds the speaker information referred to by the language switching unit 123. As described above, the translation dictionary 194 includes the word dictionary for morphological analysis referred to by the translation processing unit 125 and the translation dictionary.

FIG. 2 is a block diagram illustrating a configuration example of the direction identification unit.
The direction identification unit 122 in the speech processing apparatus 1 of the present embodiment uses the sound pressure difference of the average power in the utterance section as described above as the characteristic difference between the frames in the utterance section used for speaker identification. Therefore, the direction identification unit 122 includes a frame power calculation unit 122A, an average power calculation unit 122B, a sound pressure difference calculation unit 122C, and a speaker direction determination unit 122D, as shown in FIG.

  The frame power calculation unit 122A calculates the power of each frame in the audio data held by the audio data holding unit 191. For example, the frame power calculation unit 122A calculates the power of the frame from the amplitude of the time waveform for each frame. The frame power calculation unit 122A causes the calculation result holding unit 193 to hold the calculated power of each frame.

  The average power calculation unit 122B calculates the average power from the power of the frames included in the utterance section. The average power calculation unit 122B in the speech processing apparatus 1 according to the present embodiment is a frame that is a sound among a plurality of frames from the start frame of the utterance section detected by the utterance detection unit 121 to the current processing target frame. Calculate the average power for.

  The sound pressure difference calculation unit 122C calculates the sound pressure difference based on the average power of the utterance section calculated for each voice data. The sound pressure difference calculation unit 122C calculates the sound pressure difference DiffPow by, for example, the following equation (1).

  In Equation 1, AvePow_L and AvePow_R are the average power for the audio data acquired from the L-channel microphone 201 and the average power for the audio data acquired from the R-channel microphone 202, respectively.

  The speaker direction determination unit 122D determines the direction in which the speaker is present based on the magnitude relationship between the sound pressure difference calculated by the sound pressure difference calculation unit 122C and a predetermined threshold. When the sound pressure difference DiffPow calculated by Expression (1) is larger than the threshold value TH (when TH <DiffPow), the speaker direction determination unit 122D is a story that exists on the left side (left side of the microphone) when viewed from the stereo microphone 2. The person is speaking. On the other hand, when the sound pressure difference DiffPow calculated by the equation (1) is smaller than the threshold value −TH (when DiffPow <−TH), the speaker direction determination unit 122D is the right side (the right side of the microphone) as viewed from the stereo microphone 2. Is determined to be speaking. Furthermore, in the present embodiment, when the sound pressure difference DiffPow calculated by the equation (1) is −TH ≦ DiffPow ≦ TH, the speaker direction determination unit 122D has a plurality of speakers speaking (speaker direction). Cannot be specified).

  The determination result of the speaker direction determination unit 122D is input to the language switching unit 123. Based on the processing result of the speaker direction determination unit 122D (direction identification unit 122) and the speaker position information held in the speaker information holding unit 193, the language switching unit 123 translates the source language and the translation destination. Switch to language.

  FIG. 3 is a diagram illustrating an example of the positional relationship between the microphone and the speaker. FIG. 4 is a diagram illustrating a setting example of speaker information.

  As described above, the audio processing device 1 according to the present embodiment acquires audio data from the stereo microphone 2 including the L channel microphone 201 and the R channel microphone 202. At this time, as shown in FIG. 3, for example, the stereo microphone 2 is installed at a position between the two speakers 4A and 4B having a conversation (conversation). In addition, the stereo microphone 2 is located in the direction in which the L-channel microphone 201 is located in the direction where the first speaker 4A is present and the second speaker 4B is present as seen from the midpoint between the two microphones 201 and 202. It is installed in the direction in which the R channel microphone 202 is located.

  When the utterance language of the first speaker 4A is Japanese and the utterance language of the second speaker 4B is English, the speaker information holding unit 193 stores, for example, the speaker information shown in FIG. Hold. In the speaker information of FIG. 4, the speaker ID is an identifier for identifying the first speaker 4A and the second speaker 4B. The position of the speaker is information indicating the positions of speaker 1 and speaker 2 as viewed from stereo microphone 2 (the direction in which they exist). The utterance language is information indicating the utterance languages of the speaker 1 and the speaker 2. In FIG. 3, the speaker 1 on the left side of the microphone is the first speaker 4A, and the language of the speaker 1 is Japanese. In FIG. 3, the speaker 2 on the right side of the microphone is the second speaker 4B, and the speaking language of the speaker 2 is English. Therefore, in the speaker information 501 of FIG. 4, the speaker 1 is associated with information indicating that the position of the speaker is on the left side of the microphone and that the utterance language is Japanese. Similarly, in the speaker information 501 of FIG. 4, the speaker 2 is associated with information indicating that the position of the speaker is on the right side of the microphone and that the utterance language is English.

  When the first speaker 4A located on the left side of the microphone with the stereo microphone 2 installed as described above speaks, the audio processing device 1 is located on the left side of the microphone based on the audio data acquired from the stereo microphone 2. It is identified that the first speaker 4A is speaking. For this reason, the speech processing apparatus 1 translates the utterance content of the first speaker 4A using, for example, speech data acquired from the L channel microphone 201. When the utterance language of the first speaker 4A is Japanese and the utterance language of the second speaker 4B is English, the speech processing apparatus 1 uses the speech data in the speech data acquired from the L channel microphone 201. Is translated into Japanese, and the sentence is translated into English and output to the display device 3. Therefore, the second speaker 4B can understand the utterance contents of the first speaker 4A by reading the text (English) displayed on the display device 4.

  On the other hand, when the second speaker 4B speaks, the speech processing apparatus 1 translates the speech content of the second speaker 4B from English to Japanese using the speech data acquired from the R channel microphone 202. .

  As described above, the speech processing apparatus 1 according to the present embodiment divides each speech data acquired from the microphones 201 and 202 into frames, and identifies a speaker in the speech section of the acquired speech data. Processing to translate utterance content.

  The first audio processing unit 110 (the audio data acquisition unit 111 and the holding data control unit 112) of the audio processing device 1 performs processing for dividing the audio data into frames and holding it. The audio data acquisition unit 111 acquires audio data from each of the plurality of microphones 201 and 202. The holding data control unit 112 divides each acquired audio data into frames having a predetermined time length, and causes the audio data holding unit 191 to hold each frame in a manner in which the arrangement order according to time can be identified. The audio data acquisition unit 111 and the holding data control unit 112 continue the process of dividing the audio data into frames and holding them until receiving an input of a process end command.

  On the other hand, the process of translating the utterance content of the utterance section in the speech data is performed by the second speech processing unit 120 (the speech detection unit 121, the direction identification unit 122, the translation language switching unit 123, the speech data extraction unit 124) of the speech processing apparatus 1. The translation processing unit 125 and the output unit 126). The second voice processing unit 120 of the voice processing device 1 performs the process shown in FIG. 5 as a process of translating the utterance content of the utterance section.

FIG. 5 is a flowchart for explaining the utterance content translation processing according to the first embodiment.
As shown in FIG. 5, the process performed by the second sound processing unit 120 of the sound processing device 1 is a loop process (step S2) for performing a process after step S2 on each set of frames at the same time in a plurality of sound data. S1-S10). The second audio processing unit 120 performs the processing from step S2 on all frame sets included in the audio data, and ends the loop processing when it reaches the end of the loop processing (step S10).

  In the process for one set of frames, the second audio processing unit 120 first performs an utterance detection process (step S2) for detecting an utterance for each frame included in the set of frames. The processing in step S2 is performed by the utterance detection unit 121 of the second voice processing unit 120. The utterance detection unit 121 determines whether the processing target frame is a sound frame according to a known determination method. Further, when the processing target frame is a sound frame, the speech detection unit 121 goes back a predetermined number of frames from the frame in the past, and detects the speech based on the number of sound frames in the past frame. When the number of sound frames in a predetermined number of past frames reaches the threshold value, the utterance detection unit 121 determines (recognizes) the detection of the utterance.

  When the process of step S2 is completed, the utterance detection unit 121 continues to determine whether or not an utterance has been detected based on the processing result of the utterance detection process (step S3). When the utterance is not detected (step S3; NO), the utterance detection unit 121 ends the process for the frame that is the current processing target. Thereafter, when there is a set of unprocessed frames, the second sound processing unit 120 starts the processing from step S2 onward for the next set of frames.

  On the other hand, when an utterance is detected (step S3; YES), the utterance detection unit 121 next determines whether or not the utterance is continuing (step S4). In step S <b> 4, the utterance detection unit 121 determines whether or not the utterance is continuing based on, for example, the processing result of the previous processing in the loop processing.

  When the utterance is detected but not continued (step S4; NO), the second voice processing unit 120 next performs a direction identification process (step S5) for identifying the direction of the speaker. The process of step S5 is performed by the direction identifying unit 122 of the second sound processing unit 120. The direction identifying unit 122 goes back a predetermined number of frames in the past from the current processing target frame, and calculates the average power based on the power of the sound frame. In addition, the direction identification unit 122 calculates a sound pressure difference between utterance sections based on the average power, and identifies the direction of the speaker in the utterance section including the current processing target frame based on the sound pressure difference. When the loop processing S1 to S10 is performed using two audio signals acquired from the stereo microphone 2, the direction identification unit 122 indicates that the direction of the speaker is the left microphone, the right microphone, and the front direction (not specified). It identifies whether it is either.

  When the process of step S5 is completed, the direction identifying unit 122 continues to determine whether or not the direction of the speaker has been identified as the direction of one microphone (step S6). When loop processing S1 to S10 is performed using two audio signals acquired from the stereo microphone 2, the direction identification unit 122 identifies that the direction of the speaker is either the left microphone or the right microphone. It is determined that the direction of the speaker is identified as the direction of one microphone. When the direction of the speaker cannot be identified (step S6; NO), the direction identification unit 122 ends the process for the current processing target frame. Thereafter, when there is a set of unprocessed frames, the second sound processing unit 120 starts the processing from step S2 onward for the next set of frames.

  On the other hand, when the direction of the speaker is identified (step S6; YES), the second speech processing unit 120 next switches the translation language based on the identified direction of the speaker (step S7). The language switching unit 123 performs the process in step S7. The language switching unit 123 is based on the identification result of the direction of the speaker in the utterance section including the current processing target frame, and the speaker position information (see FIG. 4) in the speaker information holding unit 193. Set the combination of source language and target language.

  When the process of step S7 is performed or when the utterance is continuing (step S4; YES), the second voice processing unit 120 next extracts voice data to be translated (step S8). The voice data extraction unit 124 performs the process in step S8. The voice data extraction unit 124 reads, from the voice data holding unit 191, a frame (voice data) of an utterance section to be translated in the voice data including the current processing target frame.

  Next, the second voice processing unit 120 translates and outputs the utterance content of the extracted voice data (step S9). The translation processing unit 125 and the output unit 126 perform the process in step S9. The translation processing unit 125 converts speech data (speech section) into a character string according to a known analysis method, performs morphological analysis on the character string, and converts the speech data into sentences. At this time, the translation processing unit 125 refers to the morphological analysis dictionary included in the translation dictionary 194 and documents the voice data in the translation source language set in step S7. Thereafter, the translation processing unit 125 translates (converts) the document in the translation source language into the document in the translation destination language in accordance with a known translation method. When the translation processing in the translation processing unit 125 ends, the output unit 126 creates display data for visualizing the translated document data and outputs the display data to the display device 3. Thereby, the translated text of the utterance content in the utterance section including the frame that is the current processing target is displayed on the display device 3.

  When the process of step S9 is completed, the process for the current set of frames to be processed ends. Here, when there is a set of unprocessed frames, the second sound processing unit 120 starts the processing from step S2 onward for the next set of frames. In addition, when the processing after step S2 is performed on all sets of frames, the second speech processing unit 120 ends the loop processing (steps S1 to S10) and ends the utterance content translation processing.

  In the utterance detection process (step S2) according to the present embodiment, as described above, when the current processing target frame is a voiced frame, the frame is traced back to the past and the presence in a predetermined number of frames is detected. Check the number of sound frames. When the number of frames with sound reaches the threshold, the utterance detection unit 121 recognizes that the utterance has been detected. In other words, the utterance detection unit 121 sets the number of uttered frames in the frame corresponding to the utterance detection delay length d_frame including the current processing target frame to the threshold value TH even if the voicing frames are not continuous. When it arrives, it confirms (recognizes) the detection of the utterance. The utterance detection delay length d_frame corresponds to the number of frames from the start of counting the number of voiced frames, which is a condition for determining utterance detection, to the frame for which utterance detection is determined. The utterance detection process performed by the utterance detection unit 121 of this embodiment will be described with reference to FIGS. 6A and 6B.

  FIG. 6A is a flowchart (part 1) for explaining the contents of the speech detection process. FIG. 6B is a flowchart (part 2) illustrating the content of the speech detection process.

  As shown in FIGS. 6A and 6B, the utterance detection process performed by the utterance detection unit 121 according to the present embodiment is a loop process in which the processes after step S202 are performed on each of a plurality of frames included in one set of frames. (S201 to S211). The utterance detection unit 121 performs the processing from S202 on all the frames included in one set of frames, and ends the loop processing when reaching the end of the loop processing (step S211).

  In the processing for one frame, the speech detection unit 121 first determines whether or not the processing target frame is sounded, and if it is sounded, sets the value VAD representing the determination result for the frame to 1 (Step S202). When the processing target frame is not sounded, the utterance detection unit 121 sets a value VAD representing the determination result to a value different from 1 (for example, 0).

  Next, the speech detection unit 121 checks whether or not the processing target frame is VAD = 1 (sound) (step S203).

  When the processing target frame is VAD = 1 (step S203; YES), the speech detection unit 121 performs the processing of steps S204 and S205, and sets the value of the flag f_speech representing the processing result of the speech detection processing to 1 or 0. Set. In step S204, the speech detection unit 121 updates the counter vad_count that counts the number of frames with VAD = 1 to vad_count + 1, and resets the counter n_vad_count that counts the number of frames with VAD ≠ 1 to zero. In step S205, the speech detection unit 121 determines whether or not a counter vad_count that counts the number of frames with VAD = 1 is greater than a threshold value TH1. When vad_count> TH1 (step S205; YES), the utterance detection unit 121 sets the flag f_speech indicating the processing result to a value (for example, 1) indicating that the utterance has been detected, and sets the utterance detection delay length d_frame. Setting is made (step S206). In step S206, the utterance detection unit 121 sets the number of frames from the frame in which VAD = 1 to the current processing target frame to the utterance detection delay length d_frame. On the other hand, when vad_count ≦ TH1 (step S205; NO), the utterance detection unit 121 sets the flag f_speech indicating the processing result to a value (for example, 0) indicating that no utterance has been detected (step S207). .

  On the other hand, when the frame to be processed is VAD ≠ 1 (step S203; NO), the utterance detection unit 121 performs the processing of steps S208 to S210 in FIG. 6B and the value of the flag f_speech indicating the processing result of the utterance detection process. Is set to 1 or 0. In step S208, the utterance detection unit 121 updates the counter n_vad_count that counts the number of frames with VAD ≠ 1 to n_vad_count + 1. In step S209, the speech detection unit 121 determines whether or not a counter n_vad_count that counts the number of frames of VAD ≠ 1 is larger than the threshold value TH2. If n_vad_count> threshold TH2 (step S209; YES), the speech detection unit 121 next resets a counter vad_count that counts the number of frames with VAD = 1 to 0. After step S209, the utterance detection unit 121 sets the flag f_speech representing the processing result to a value (f_speech = 0) indicating that no utterance has been detected (step S207). On the other hand, when n_vad_count ≦ threshold TH2 (step S209; NO), the utterance detection unit 121 next sets a flag f_speech indicating a processing result to a value (f_speech = 1) indicating that the utterance has been detected. An utterance detection delay length d_frame is set (step S206).

  When step S206 or S207 is completed, the processing for the current processing target frame is completed, and the processing performed by the utterance detection unit 121 reaches the end of the loop processing (step S211). Here, when there is an unprocessed frame, the utterance detection unit 121 starts processing from step S202 onward for the next frame. Further, when the processing from step S202 is performed on all frames, the utterance detection unit 121 ends the loop processing and ends the utterance detection processing.

  As described above, in the utterance detection process according to the present embodiment, the utterance detection unit 121 determines the detection of the utterance when the number of sound frames in the predetermined number of consecutive frames reaches the threshold value TH1. Furthermore, the utterance detection unit 121 sets the number of frames from the frame that starts counting the number of voiced frames, which is a condition for determining the detection of utterance, to the frame that determines the detection of utterance as the utterance detection delay length d_frame, A frame that goes back in the past by the utterance detection delay length d_frame from the frame in which the utterance is detected is set as the start position of the utterance section. Therefore, according to the present embodiment, it is possible to correctly set the start position of the utterance section including the frame in which the utterance detection is confirmed. As a result, for example, when an utterance section is extracted from audio data, it is possible to suppress the beginning of words due to a difference between the utterance start position in the audio data and the frame in which the utterance is detected. Further, according to the utterance detection process described above, since the frame determined to be sound is the start position of the utterance section, it is possible to reduce the noise section before the utterance included in the utterance section.

  Next, the content of the direction identification process (step S5) in the flowchart of FIG. 5 will be described with reference to FIG.

FIG. 7 is a flowchart for explaining the contents of the direction identification processing.
The direction identification process is performed by the direction identification unit 122. The direction identifying unit 122 first performs a loop process (S501 to S506) for calculating the average power of the speech section including the frame for each of a plurality of frames included in the set of one frame.

  In the loop processing for calculating the average power, the direction identification unit 122 first determines whether or not the processing target frame is VAD = 1 (sound frame) (step S502). The power calculation unit 122A of the direction identification unit 122 performs the process of step S502. When the processing target frame is VAD = 1 (step S502; YES), the power calculation unit 122A calculates the frame power of the processing target frame (step S503) and causes the calculation result holding unit 192 to hold it. When the processing target frame is a frame of L channel audio data of the two audio data acquired from the stereo microphone 2, the power calculation unit 122A calculates the power FramePow_L by the following equation (2-1), for example. To do. When the processing target frame is a frame of R channel audio data out of the two audio data acquired from the stereo microphone 2, the power calculation unit 122A uses, for example, the power FramePow_R according to the following equation (2-2). Is calculated.

  S_L in Expression (2-1) and s_R in Expression (2-2) are audio data for one frame of the L channel and audio data for one frame of the R channel, respectively. Moreover, the variable n of Formula (2-1) and Formula (2-2) is the number of samples.

  On the other hand, when the frame to be processed is VAD ≠ 1 (step S502; NO), the power calculation unit 122A sets the power of the frame to be processed to 0 (step S504) and causes the calculation result holding unit 192 to hold it.

  When the process of step S503 or S504 is completed, the direction identification unit 122 calculates the average power for the frame of VAD = 1 among the frames for the speech detection delay length d_frame in the audio data including the processing target frame. (Step S505). The process of step S505 is performed by the average power calculation unit 122B of the direction identification unit 122. The average power calculation unit 122B calculates the average power by reading the power for the frame of VAD = 1 among the frames corresponding to the speech detection delay length d_frame held by the calculation result holding unit 192. When the processing target frame is a frame of L channel audio data out of the two audio data acquired from the stereo microphone 2, the power calculation unit 122A calculates the average power AvePow_L by the following equation (3-1), for example. calculate. When the processing target frame is a frame of R channel audio data out of the two audio data acquired from the stereo microphone 2, the power calculation unit 122A, for example, calculates the average power by the following equation (3-2). Calculate AvePow_R.

  The value M in the equations (3-1) and (3-2) is the number of frames with VAD = 1 among the frames with the speech detection delay length d_frame.

When the process of step S505 is completed, the process for the current processing target frame is completed, and the process performed by the direction identification unit 121 reaches the end of the loop process (step S505). Here, when there is an unprocessed frame, the direction identification unit 122 (the power calculation unit 122A and the average power calculation unit 122B) starts the processing after step S502 for the next frame. In addition, when the processing after step S502 is performed for all the frames, the direction identification unit 122 ends the loop processing, and then the sound pressure difference based on the average power calculated in the loop processing (steps S501 to S506). Is calculated (step S507). The process of step S507 is performed by the sound pressure difference calculation unit 122C of the direction identification unit 122. When the set of frames to be processed is an L-channel audio data frame and an R-channel audio data frame acquired from the stereo microphone 2, the sound pressure difference calculation unit 122C calculates the sound pressure difference DiffPow according to the above equation (1). Next, the direction identifying unit 122 identifies the direction of the speaker who is speaking based on the calculated sound pressure difference DiffPow and the threshold value TH3 (step S508). The process of step S508 is performed by the speaker direction determination unit 122D. When the sound pressure difference DiffPow is calculated by the equation (1), DiffPow> 0 when AvePow_L> AvePow_R, and DiffPow <0 when AvePow_L <AvePow_R. Therefore, when the threshold TH3> 0, the speaker direction determination unit 122D identifies that the speaker is on the left side of the microphone if TH3 <DiffPow, and the speaker is on the right side if DiffPow <−TH3. Identify.

  When the process of step S508 is completed, the direction identification unit 122 transmits the speaker direction identification result to the language switching unit 123 and ends the direction identification process.

  As described above, the direction identification unit 122 according to the present embodiment has a characteristic of a frame that is sound among frames that are determined based on the frame in which the utterance is detected and the utterance detection delay length d_frame ( The direction of the speaker is identified based only on the waveform. Therefore, when a frame with a small amplitude (a frame with VAD ≠ 1) is included in the speech section, the direction identification unit 122 excludes the frame with the small amplitude and the frame with a small signal-to-noise ratio and excludes the speaker's It becomes possible to identify the direction. Therefore, according to the present embodiment, it is possible to reduce the error in identifying the direction of the speaker due to the frame of VAD ≠ 1 included in the speech section.

FIG. 8 is a diagram illustrating processing performed by the first sound processing unit.
FIG. 8A shows an example of the audio data 6 acquired from the stereo microphone 2 and a data processing unit (frame) when the audio data holding unit 191 holds the data. The audio data 6 collected from the stereo microphone 2 is, for example, audio data in Pulse Code Modulation (PCM) format. In this case, the audio data 6 is a stereo signal obtained by using one frame of audio data collected by the L channel microphone 201 and one frame of audio data collected by the R channel microphone 202 as a set of frame data. Output from the microphone 2. In the first sound processing unit 110 in the sound processing apparatus 1 of the present embodiment, the sound data 6 is converted into frame data 601, 602, and 603 of a set of L channel frames and R channel frames collected at the same time. , ...

FIG. 8B shows a diagram for explaining the amount of frame data held in the audio data holding unit 191. For example, the audio data holding unit 191 holds D _max frame data (a set of L channel frames and R channel frames). Here, the maximum value D _{max of the} number of frames to be held is not less than a value that can be set as the speech detection delay length d_frame. In addition, the holding data control unit 112 of the first audio processing unit 110 causes the audio data holding unit 191 to hold a new frame (h + D _max- th frame) 614 of the frames 611 to 613 that are already held. The earliest (earliest) frame (h-th frame) 611 is deleted.

  Note that the audio data 6 illustrated in FIG. 8A is only an example of audio data acquired by the first audio processing unit 110. The audio data acquired by the first audio processing unit 110 may be independent of the audio data collected by the L-channel microphone 201 and the audio data collected by the R-channel microphone 202. Further, the audio data holding method shown in FIG. 8B is merely an example. The first audio processing unit 110 may cause the audio data holding unit 191 to sequentially hold all frames from the first frame in the acquired audio data.

FIG. 9 is a diagram illustrating an example of a method for determining whether or not a frame is sounded.
FIG. 9 is a diagram for explaining Voice Activity Ditection (VAD), which is one of the methods for determining whether or not a frame in audio data is a voiced frame. The section from time t0 to t1 and the section from time t4 to t5 in the time waveform 701 of the sound data shown in FIG. 9 are non-speech sections (sections with VAD = 0) that do not include significant speech, respectively. . On the other hand, a section from time t1 to t4 in the time waveform is a voice section including a significant voice (section with VAD = 1).

  Further, when the time waveform 701A in the section from time t2 to t3 in the voice section of the voice data is seen, for example, as shown in FIG. 9, the time waveform has periodicity.

Whether or not the frame is a voiced frame by VAD is determined by using the above-described feature in the waveform of the voice section to determine whether or not the processing target frame is a voiced frame. In other words, the speech detection unit 121 in the speech processing device 1 according to the present embodiment determines whether the processing target frame depends on whether the processing target frame satisfies the following conditions 1 and 2 in the process of step S202, for example. It is determined whether or not there is sound.
(Condition 1) The fluctuation range of the amplitude level is larger than the threshold value.
(Condition 2) The time waveform has periodicity.

FIG. 10 is a diagram illustrating an example of a processing result of the speech detection processing.
When the second speech processing unit 120 of the speech processing apparatus 1 of the present embodiment performs speech detection processing on speech data having the waveform 702 shown in FIG. 10, for example, a processing result such as a table 502 is obtained. It is done. In the table 502, FN is a frame number for identifying a frame to be processed, and VAD is a value indicating whether each frame is a sound frame. Further, vad_count is a counter value that counts the number of frames with sound, and n_vad_count is a value of a counter that counts the number of frames without sound. Further, f_speech is a flag value indicating whether or not utterance detection is confirmed, and d_frame is a value indicating the utterance detection delay length.

  The position and the horizontal width of each square of the frame number FN in the table 502 correspond to the section delimited by the broken line in the waveform 702. That is, the amplitude of the waveform in the frame of frame number FN = h is very small compared to the amplitude of the waveform in the frame of frame number FN> h. For this reason, in the speech detection process for the frame of frame number h, VAD = 0 in step S202. At this time, vad_count and n_vad_count for the frame with frame number h are, for example, 0 and MA, respectively. Here, MA is an integer that is, for example, 1 or more and smaller than the threshold value TH2 used in the determination in step S209. Since the frame with frame number h is VAD = 0, f_speech = 0, and the speech detection delay length d_frame is indefinite.

  Next, the utterance detection unit 121 performs an utterance detection process for the frame with frame number h + 1. The waveform in the frame with the frame number h + 1 has a larger amplitude than the frame with the frame number h, and a periodicity is observed in the fluctuation of the amplitude. For this reason, the frame of frame number h + 1 is VAD = 1 in step S202. Therefore, as shown in the table 502, the utterance detection unit 121 sets the values of the counters vad_count and n_vad_frame to 1 and 0, respectively, in the process for the frame with the frame number h + 1 (step S204). If the threshold value TH1 used in the determination in step S205 is 4, the flag f_speech for the frame with frame number h + 1 is 0. On the other hand, the utterance detection delay length d_frame is 1.

  Thereafter, when the same processing is repeated for frames after frame number h + 2, the processing result as shown in the table 502 is obtained. In the processing result shown in the table 502, all the frames after the frame number h + 1 are VAD = 1, and the value of vad_count is increased by 1. Therefore, the detection of the utterance is determined in the frame of frame number h + 4 which is the fourth frame counted from the frame of frame number h + 1 changed from VAD = 0 to VAD = 1, and f_speech = 1. Further, when performing the speech detection process for the frame of frame number h + 4, the speech detection delay length d_frame = 4. Therefore, the second speech processing unit 120 (speech processing device 1) uses the frame number h + 4 of the frame for which speech detection is confirmed and the speech detection delay length d_frame = 4 as the speech interval for the frame of frame number h + 1. Recognized as the first frame. As a result, it is possible to include in the utterance section three frames with frame numbers h + 1 to h + 3 prior to the frame with frame number h + 4 that detected the utterance, and the utterance contents of the three frames with frame numbers h + 1 to h + 3 are included. It becomes possible to translate.

FIG. 11 is a diagram illustrating another example of the processing result of the speech detection process.
When the second speech processing unit 120 of the speech processing apparatus 1 according to the present embodiment performs speech detection processing on speech data having the waveform 703 shown in FIG. 11, for example, a processing result such as a table 503 is obtained. It is done. In the table 503, FN is a frame number for identifying a frame to be processed, and VAD is a value indicating whether each frame is a sound frame. Further, vad_count is a counter value that counts the number of frames with sound, and n_vad_count is a value of a counter that counts the number of frames without sound. Further, f_speech is a flag value indicating whether or not an utterance has been detected, and d_frame is a value indicating the utterance detection delay length.

  The position and width of each square of the frame number FN in the table 503 correspond to the section delimited by the broken line in the waveform 703. That is, the amplitude of the waveform in the frame of frame number FN = k is very small compared to the amplitude of the waveform in the frame of frame number FN> k. For this reason, in the speech detection process for the frame of frame number k, VAD = 0 in step S202. At this time, the values of the counters vad_count and n_vad_count in the process for the frame with the frame number k are, for example, 0 and MB, respectively. Further, since the frame with frame number k is VAD = 0, the flag f_speech is 0, and the speech detection delay length d_frame is indefinite.

  Next, the utterance detection unit 121 performs an utterance detection process for the frame with frame number k + 1. The waveform in the frame of frame number k + 1 has a larger amplitude than the frame of frame number k, and a periodicity is seen in the fluctuation of the amplitude. For this reason, the frame of frame number k + 1 is VAD = 1 in step S202. Accordingly, as shown in the table 503, the utterance detection unit 121 sets the values of the counters vad_count and n_vad_frame to 1 and 0, respectively, in the process for the frame with the frame number k + 1 (step S204). When the threshold value TH1 used in the determination in step S205 is 4, the flag f_speech for the frame with frame number h + 1 is 0 and the speech detection delay length d_frame is 1.

  Thereafter, when the threshold TH2 used for the determination in step S209 is set to 3 and the same processing is repeated for frames after the frame number k + 2, the processing result as shown in the table 503 is obtained. That is, VAD = 1 in the frames of frame numbers k + 1, k + 2, k + 5, and k + 8, and the counter vad_count is 4 in the frame of frame number k + 8. For this reason, in the example shown in FIG. 11, the flag f_speech becomes 1 in the frame of frame number k + 8, and the detection of the utterance is confirmed. Further, since the count of the utterance detection delay length d_frame starts when the process for the frame with the frame number k + 1 is performed, the utterance detection delay length d_frame with the process for the frame with the frame number k + 8 is 8. Therefore, the second speech processing unit 120 (speech processing device 1) uses the frame number k + 8 of the frame in which the speech is detected and the speech detection delay length d_frame = 8 as the start of the speech section. Recognize as a frame. As a result, it is possible to include three frames of frame numbers k + 1, k + 2, and k + 5 before the frame of frame number k + 8 in which the utterance is detected in the utterance section, and translate the utterance contents of the three frames. Is possible.

FIG. 12 is a diagram for explaining an average power calculation method in the direction identification processing.
When the second sound processing unit 120 of the sound processing apparatus 1 according to the present embodiment performs the process of calculating the frame power on the sound data having the waveform 704 illustrated in FIG. Results are obtained. In the table 503, FN is a frame number for identifying a frame to be processed, and VAD is a value indicating whether each frame is a sound frame. The frame power is a value indicating the power of each frame calculated by the processes in steps S502 to S504 in FIG.

  The position and width of each square of the frame number FN in the table 504 correspond to the section delimited by the broken line in the waveform 704. That is, the amplitude of the waveform in the frame of frame number FN ≦ j−6 is very small compared to the amplitude of the waveform in the frame of frame number j + 5. For this reason, in the speech detection process for the frames with the frame numbers j-10 to j-6, VAD = 0 in step S202. Similarly, frames with frame numbers j−2 and j−1 having a smaller waveform amplitude than VAD = 1 such as frame number j + 5 have VAD = 0. Here, if the thresholds TH1 and TH2 in the utterance detection process are 4 and 3, respectively, the utterance detection unit 121 detects the utterance when the process for the frame with the frame number j is performed. Therefore, the start position of the utterance section is the frame of frame number j-5.

  At this time, the direction identification unit 122 in the second audio processing unit 120 calculates the power only for the frame with VAD = 1 by the above formulas (2-1) and (2-2), and the frame of VAD = 0. The power is 0 (steps S502 to S504). Furthermore, the direction identification unit 122 calculates the average power for the power of the frame of VAD = 1 among the frames for the speech detection delay length d_frame including the frame of the frame number j (step S505). That is, when the frame with the frame number j in the example of FIG. 12 is the processing target, the direction identification unit 122 calculates the power P1 calculated from each of the frames with the frame numbers j-5, j-4, j-3, and j. , P2, P3, and P4 are calculated. Thus, by calculating the average power based only on the power of the VAD = 1 frame among the frames corresponding to the utterance detection delay length d_frame, the non-speech section without the utterance section or the section with a small S / N ratio is obtained. It is possible to calculate the average power excluding the power.

FIG. 13 is a diagram for explaining the relationship between the sound pressure difference and the direction of the speaker.
FIG. 13A schematically shows the relationship between the direction of the sound source (speaker) seen from the two microphones 201 and 202 included in the stereo microphone 2 and the sound pressure in the audio data output from each microphone. Is shown.

  As shown in FIG. 13A, when the stereo microphone 2 is installed such that the R channel microphone 202 is positioned on the right side when viewed from the L channel microphone 201, the left side of the L channel microphone 201 is the left side of the microphone. Become. In this case, the right side of the R channel microphone 202 is the right side of the microphone.

  When a speaker at a position on the left side of the microphone speaks, the distance from the speaker to the L-channel microphone 201 is shorter than the distance from the speaker to the R-channel microphone 202. Therefore, when a speaker in the area 801 on the left side of the microphone speaks, the voice data acquired from the L-channel microphone 201 has higher power than the voice data acquired from the R-channel microphone 202. On the other hand, when a speaker who is present in the area 803 on the right side of the microphone speaks, the voice data acquired from the R channel microphone 202 has higher power than the voice data acquired from the L channel microphone 201. On the other hand, when a speaker existing in the area 802 in the front direction when viewed from the microphones 201 and 202 speaks, the power of the two audio data is higher than when a speaker existing in the areas 801 and 803 speaks. The difference is smaller.

  Based on the above points, in the present embodiment, for example, as in the table 505 shown in FIG. 13B, the user speaks based on the relationship between the sound pressure difference calculated by the expression (1) and the threshold value. Identify the direction of the speaker. The sound pressure difference DiffPow calculated by the equation (1) is DiffPow> 0 when a speaker existing on the left side of the microphone is speaking, and DiffPow <0 when a speaker existing on the right side of the microphone is speaking. Become. Therefore, in the table 505, the correspondence relationship between the sound pressure difference and the direction of the speaker is classified into three types according to the threshold value TH3 (> 0). In the table 505, an output value indicating a speaker direction identification result is registered. That is, when processing is performed on a frame where a speaker located on the left side of the microphone is speaking, an output value 1 is output, and processing is performed on a frame where a speaker located on the right side of the microphone is speaking. If so, output value -1 is output. Further, when the absolute value of the sound pressure difference is equal to or smaller than the absolute value of the threshold value TH3, it is possible to identify (specify) whether the speaker speaking is left or right of the microphone. An output value 0 indicating that there was no output is output.

FIG. 14 is a diagram illustrating a translation language switching method.
FIG. 14A again shows the same diagram as FIG. As described above, the audio processing device 1 according to the present embodiment acquires audio data from the stereo microphone 2 including the L channel microphone 201 and the R channel microphone 202. At this time, as shown in FIG. 14, for example, the stereo microphone 2 is installed at a position between the two speakers 4A and 4B having a conversation (conversation). In addition, the stereo microphone 2 is located in the direction in which the L-channel microphone 201 is located in the direction where the first speaker 4A is present and the second speaker 4B is present as seen from the midpoint between the two microphones 201 and 202. It is installed in the direction in which the R channel microphone 202 is located. Further, it is assumed that the first speaker 4A has an utterance language of Japanese, and the second speaker 4B has an utterance language of English. At this time, the speaker information holding unit 193 stores, for example, the speaker position information 501 shown in FIG.

  When the first speaker 4A speaks in this state, the speech processing apparatus 1 according to the present embodiment performs the above processes and determines that the speaker located on the left side of the microphone is speaking. In this case, the speech processing apparatus 1 uses the L-channel microphone based on the speaker position information 501 and an output value (see FIG. 13B) indicating that the direction of the speaker is to the left of the microphone. The voice data acquired from 201 is extracted as voice data for translation. Thereafter, the voice processing device 1 translates the voice data (speech section) acquired by the translation processing unit 125. Note that, as described above, the sound processing device 1 of the present embodiment determines whether or not there is sound for each frame, and when an utterance is detected in a certain frame, from the frame in which the utterance is detected in the past Go back and set the start position of the utterance section. For this reason, in the speech processing device 1 according to the present embodiment, it is possible to include the frame immediately after the first person 4A starts speaking in the speaking section, and to prevent the beginning of words at the time of translation.

  Further, as shown in FIG. 14B, when the second speaker 4B starts speaking, the speech processing apparatus 1 according to the present embodiment performs the above-described processes and the speaker on the right side of the microphone Determine that you are speaking. In this case, the speech processing apparatus 1 translates speech data acquired from the R channel microphone 202 based on the speaker position information 501 and an output value indicating that the direction of the speaker is to the right of the microphone. Extract as voice data. Also at this time, the speech processing apparatus 1 according to the present embodiment sets the start position of the utterance section retroactively from the frame in which the utterance of the second speaker 4B is detected. Therefore, even when the speaker changes from the first person 4A to the second person 4B, the frame immediately after the second person 4B starts speaking can be included in the speech section, and the beginning of the translation It becomes possible to prevent cutting.

  Furthermore, in the present embodiment, when identifying the direction of the speaker, the voiced frame (frame with VAD = 1) of the frames included in the utterance section is traced back from the frame in which the utterance is detected. Identify the speaker's direction based on the average power. That is, in this embodiment, the direction of the speaker is identified by excluding the section with a small amplitude and the section with a small SN ratio included in the speech section. For this reason, it becomes possible to reduce an error in identifying the direction of the speaker due to the fact that the section where the speaker is not speaking is included in the speech section.

  Note that the flowcharts of FIGS. 5, 6A, 6B, and 7 are merely examples of processing performed by the second audio processing unit 120 in the audio processing device 1 of the present embodiment. The processing performed by the second audio processing unit 120 according to the present embodiment is not limited to the procedures in FIGS. 5, 6A, 6B, and 7, and can be changed as appropriate. For example, the process of step S202 for determining whether or not the frame is a sound frame is not limited to the above VAD, and whether or not the frame is sound is determined based on the power or SN ratio of the frame to be processed. Also good.

  Further, the combination of the translation source language and the translation destination language in the translation processing performed by the speech processing apparatus 1 according to the present embodiment is not limited to the above-described combination of English and Japanese, and can be changed as appropriate. Further, the combination of the translation source language and the translation destination language can be selected from a plurality of combinations.

  Furthermore, the audio data acquired by the audio processing device 1 according to the present embodiment is not limited to 2-channel audio data collected by the stereo microphone 2, but is audio data collected by two independent monaural microphones. Also good. Further, the direction of the speaker as viewed from the microphone is not limited to the left-right direction, but may be the front-rear direction.

  Moreover, the speech processing apparatus 1 according to the present embodiment may output the translation result to a speaker, a receiver, or the like instead of outputting the translation result to the display device 3.

  In addition, in the speech processing apparatus 1 according to the present embodiment, after identifying the direction of the speaker in the utterance section, the speech section is translated based on the identification result. However, the speech processing apparatus 1 is not limited to the translation processing, and may perform other speech processing based on the identification result of the direction of the speaker in the speech section. For example, the speech processing apparatus 1 may perform speech processing that summarizes the speech content of each speaker in the speech data for each speaker based on the identification result of the speaker direction in the speech section.

<Second Embodiment>
In the present embodiment, another example of audio processing performed by the second audio processing unit 120 in the audio processing device 1 described in the first embodiment will be described. As shown in FIG. 1, the speech processing apparatus 1 according to the present embodiment includes a first speech processing unit 110, a second speech processing unit 120, and a speaker information setting unit 130. The first audio processing unit 110 includes an audio data acquisition unit 111 and a retained data control unit 112. The second voice processing unit 120 includes an utterance detection unit 121, a direction identification unit 122, a language switching unit 123, a voice data extraction unit 124, a translation processing unit 125, and an output unit 126. The speech processing apparatus 1 according to the present embodiment also includes a speech data holding unit 191, a calculation result holding unit 192, a speaker information holding unit 193, and a translation dictionary 194.

  FIG. 15 is a flowchart for explaining the utterance content translation processing according to the second embodiment.

  The process performed by the second sound processing unit 120 in the sound processing apparatus 1 according to the present embodiment performs the processes from step S2 onward for each set of frames at the same time in a plurality of sound data, as shown in FIG. Loop processing (steps S1 to S10) is performed. The second audio processing unit 120 performs the processing from step S2 on all frame sets included in the audio data, and ends the loop processing when it reaches the end of the loop processing (step S10).

  In the utterance content translation process according to the present embodiment, when the utterance is detected in the processes of steps S2 and S3 (step S3; YES), the direction identification process (step S5) is performed without performing the determination of step S4. Do. That is, in the utterance content translation process in this embodiment, when the utterance is detected, the direction identification process is executed each time regardless of whether the utterance is continued or not. Identify the direction. And when the direction of a speaker is identified (step S6; YES), the 2nd audio | voice processing part 120 performs the process of step S7-S9 demonstrated in 1st Embodiment, and the translation with respect to the acquired audio | voice data The result is output to the display device 3 or the like.

  FIG. 16 is a diagram illustrating an example of translation processing that can occur in the audio processing according to the first embodiment.

  The four pieces of audio data 620 to 623 shown in FIG. 16 indicate the same section in one piece of audio data classified from different viewpoints. The uppermost audio data 620 shows the actual speech situation in the acquired audio data. The section from time t0 to t1 and the section from time t5 to t6 in the acquired voice data 620 are non-speaking sections where no one is speaking. Further, in the audio data 620, a speaker on the right side of the microphone speaks in a section from time t1 to t3, and a speaker on the left side of the microphone speaks in a section from time t4 to t5. Further, in the voice data 620, both speak in the section from time t3 to t4.

  The audio data 621 below the audio data 620 shows an utterance interval and a non-utterance interval when the utterance detection process is executed on the audio data 620. That is, the section from time t0 to t1 and the section from time t5 to t6 are non-speech sections, and the section from time t1 to t5 is an utterance section.

  The audio data 622 below the audio data 621 indicates a section in which the direction identification process is executed. In the first embodiment, as shown in FIG. 5, when the utterance is detected (step S3; YES) and the utterance is not continuing (step S4; NO), the direction identification process is performed. Therefore, the section in which the direction identification process is executed is a section from time t1 to t2 (<t3) immediately after the start of the speech section.

  Audio data 623 shown below the audio data 622 indicates a section in which translation processing is executed. In the first embodiment, as shown in FIG. 5, when the utterance is continuing (step S4; YES), the direction identification process (step S5) or the like is omitted and the translation process is performed. For this reason, in the speech processing device 1 according to the first embodiment, from the time t1 when the speaker on the right side of the microphone starts speaking until the time t5 when the speaker on the left side of the microphone ends speaking, Perform translation without changing the combination with the target language.

  As described above, in the speech processing according to the first embodiment, if there is a section in which both speakers speak between two sections having different speaker directions in the speech data, the source language and the translation destination An error may occur in combination with the language, and an incorrect translation result may be output.

  On the other hand, in the audio processing according to the present embodiment, as described above, the direction of the speaker is identified each time by performing the direction identification process regardless of whether or not the utterance is continued. Therefore, when the audio processing according to the present embodiment is performed on the audio data 620 in FIG. 16, a translation process as shown in FIG. 17 is performed.

FIG. 17 is a diagram illustrating an example of translation processing by speech processing according to the second embodiment.
The voice data 620 shown in FIG. 17 shows the actual speech situation in the acquired voice data. The section from time t0 to t1 and the section from time t5 to t6 in the acquired voice data 620 are non-speaking sections where no one is speaking. Further, in the audio data 620, a speaker on the right side of the microphone speaks in a section from time t1 to t3, and a speaker on the left side of the microphone speaks in a section from time t4 to t5. Further, in the voice data 620, both speak in the section from time t3 to t4.

  Audio data 621 shown below the audio data 620 indicates an utterance interval and a non-utterance interval when the utterance detection process is executed on the audio data 620. That is, the section from time t0 to t1 and the section from time t5 to t6 are non-speech sections, and the section from time t1 to t5 is an utterance section. Up to this point, the processing is the same as that described in the first embodiment.

  The audio data 625 shown below the audio data 621 indicates a section in which the direction identification process is executed. In the second embodiment, as shown in FIG. 15, when an utterance is detected (step S3; YES), direction identification processing is performed each time. Therefore, the section in which the direction identification process is executed is a section from the start time t1 of the utterance section to the end time t5 of the utterance section.

  The audio data 626 shown below the audio data 622 indicates a section in which translation processing is executed. A section from time t1 to time t3 in the audio data 626 is a section in which a speaker on the right side of the microphone is speaking. For this reason, in the speech processing apparatus 1 of the present embodiment, the speech processing of the section from time t4 to time t5 in the speech data 626 is performed using the speech language of the speaker on the right side of the microphone as the translation source language. On the other hand, a section from time t4 to t5 in the audio data 626 is a section in which a speaker on the left side of the microphone is speaking. For this reason, in the speech processing apparatus 1 according to the present embodiment, the speech processing of the section from time t4 to t5 in the speech data 626 is performed using the speech language of the speaker on the left side of the microphone as the translation source language.

  On the other hand, the section from time t3 to t4 is a section in which both are speaking. For this reason, in the speech processing apparatus 1 according to the present embodiment, in the section from the time t3 to the time t4, the direction of the speaker cannot be specified as either the left microphone or the right microphone. For this reason, in the speech processing apparatus 1 of the present embodiment, the section from the time t3 to the time t4 in the speech data 626 is not translated.

  As described above, in the utterance content translation processing according to the present embodiment, even when the utterance is continued and the direction of the speaker is changed in the middle, the translation source is changed at the timing when the direction of the speaker is changed. It becomes possible to switch the combination of language and target language. Further, if the direction of the speaker cannot be identified due to reasons such as both speaking while the utterance continues, the speech processing apparatus 1 according to the present embodiment stops the translation. Therefore, according to the present embodiment, it is possible to correctly identify the direction of the speaker in the utterance section. Therefore, it is possible to switch the combination of the translation source language and the translation destination language based on the identification result of the speaker direction, and to start and stop the translation at an appropriate timing.

<Third Embodiment>
In the present embodiment, an example will be described in which translation processing performed by the speech processing apparatus 1 according to the first embodiment and the second embodiment is performed by a server apparatus different from the speech processing apparatus 1.

  FIG. 18 is a block diagram illustrating a functional configuration of the voice processing device and the server device according to the third embodiment.

  As shown in FIG. 18, the speech processing apparatus 1 of the present embodiment includes a first speech processing unit 110, a second speech processing unit 120, a speaker information setting unit 130, and a communication unit 140. . The first audio processing unit 110 includes an audio data acquisition unit 111 and a retained data control unit 112. The second voice processing unit 120 includes an utterance detection unit 121, a direction identification unit 122, a language switching unit 123, and a voice data extraction unit 124. In addition, the speech processing apparatus 1 according to the present embodiment includes a speech data holding unit 191, a calculation result holding unit 192, and a speaker information holding unit 193.

  The first speech processing unit 110 and the speaker information setting unit 130 in the speech processing apparatus 1 of the present embodiment each have the functions described in the first embodiment. On the other hand, the second speech processing unit 120 in the speech processing apparatus 1 of the present embodiment is different from the second speech processing unit 120 in the first embodiment and the second embodiment in that the translation processing unit 125 and the output unit. 126 is omitted. That is, the second speech processing unit 120 in the speech processing apparatus 1 according to the present embodiment performs processing up to extracting speech data to be translated among the processing described in the first embodiment or the second embodiment.

  The communication unit 140 communicates with the server device 9 via a communication network (not shown) such as the Internet. For example, the communication unit 140 transmits a translation request including the audio data extracted by the audio data extraction unit 124, the translation source language, and the translation destination language to the server 9. Further, the communication unit 140 outputs the translation result received from the server 9 to the display device 3 or the like.

  The server device 9 performs a translation process on the speech data received from the speech processing device 1 and returns the translation result to the speech processing device 1. The server device 9 includes a translation processing unit 910, a communication unit 920, and a translation dictionary 991.

  The translation processing unit 910 refers to the translation dictionary 991, converts the voice data into a sentence in the translation source language, and then translates (converts) the document into a sentence in the translation destination language. The translation processing unit 910 and the translation dictionary 991 of the server device 9 correspond to the translation processing unit 125 and the translation dictionary 194 in the speech processing device 1 described in the first embodiment, respectively.

  The communication unit 920 performs processing for receiving a translation request from the speech processing device 1 and processing for returning the result of translation processing by the translation processing unit 910 to the speech processing device 1.

  As in the present embodiment, the processing load on the voice processing device 1 can be reduced by performing the translation processing of the voice data on the server device 3 different from the voice processing device 1. Further, the translation processing unit 910 and the translation dictionary 991 can be shared by the plurality of speech processing devices 1 by performing the translation processing in the server device 3 different from the speech processing device 1. For this reason, for example, the translation dictionary can be easily maintained and updated to the latest version. In addition, it is possible to suppress the compression of the capacity of the storage unit (not shown) of the speech processing device 1 by holding the translation dictionary in the speech processing device 1.

  In the present embodiment, an example in which the translation processing is performed in the server device 9 has been described. However, the server device 9 is not limited thereto, and performs other speech processing in which speech data and a speaker of the speech data are associated with each other. You may do it.

  The audio processing device 1 according to each of the above embodiments can be realized by a computer and a program executed by the computer. Hereinafter, the sound processing apparatus 1 realized by a computer and a program will be described with reference to FIG.

FIG. 19 is a diagram illustrating a hardware configuration of a computer.
As shown in FIG. 19, the computer 15 includes a processor 1501, a main storage device 1502, an auxiliary storage device 1503, an input device 1504, an output device 1505, an input / output interface 1506, a communication control device 1507, and a medium. A driving device 1508. These elements 1501 to 1508 in the computer 15 are connected to each other by a bus 1510 so that data can be exchanged between the elements.

  The processor 1501 is a central processing unit (CPU), a micro processing unit (MPU), or the like. The processor 1501 controls the overall operation of the computer 15 by executing various programs including an operating system. In addition, the processor 1501 executes a speech processing program including, for example, the utterance content translation processing illustrated in FIG. 5 or 15. When the computer 15 is operated as the speech processing apparatus 1 according to the third embodiment, the processor 1501 executes a program in which the translation process in step S9 in the flowchart shown in FIG. 5 or FIG. 15 is omitted.

  The main storage device 1502 includes a read only memory (ROM) and a random access memory (RAM) not shown. In the ROM of the main storage device 1502, for example, a predetermined basic control program read by the processor 1501 when the computer 15 is started is recorded in advance. On the other hand, the RAM of the main storage device 1502 is used as a working storage area as necessary when the processor 1501 executes various programs. The RAM of the main storage device 1502 can be used as, for example, the voice holding unit 191, the calculation result holding unit 192, and the speaker information holding unit 193 illustrated in FIGS. 1 and 19.

  The auxiliary storage device 1503 is a storage device having a larger capacity than the RAM of the main storage device 1502, and includes, for example, a hard disk drive (HDD) and a non-volatile memory (Solid State Drive (SSD)) such as a flash memory. ) Etc. The auxiliary storage device 1503 can be used to store various programs executed by the processor 1501 and various data. The auxiliary storage device 1503 stores, for example, a speech processing program including the utterance content translation processing shown in FIG. 5 or FIG. 15 or a program in which the translation processing in step S9 in the flowchart shown in FIG. 5 or FIG. Is available. Further, the auxiliary storage device 1503 can be used as, for example, the voice holding unit 191, the calculation result holding unit 192, and the speaker information holding unit 193 illustrated in FIGS. 1 and 19. Further, the auxiliary storage device 1503 can be used as a storage unit for storing the translation dictionary 194 shown in FIG.

  The input device 1504 is, for example, a keyboard device or a touch panel device. When an operator (user) of the computer 15 performs a predetermined operation on the input device 1504, the input device 1504 transmits input information associated with the operation content to the processor 1501. The input device 1504 can be used, for example, to input a command for starting voice processing, a command related to other processing that can be executed by the computer 15, setting speaker information, and the like.

  The output device 1505 is, for example, a display device such as a liquid crystal display device or an audio output device such as a speaker or a receiver. The output device 1505 can be used to display and generate a translation processing result (translated text).

  The input / output interface 1506 connects the computer 15 and other electronic devices. The input / output interface 1506 includes, for example, a universal serial bus (USB) standard connector. The input / output interface 1506 can be used for connection between the computer 15 and the plurality of microphones 201 and 202, for example.

  The communication control device 1507 is a device that connects the computer 15 to a network such as the Internet and controls various communications between the computer 15 and other communication devices via the network. The communication control device 1507 can be used, for example, for communication between the computer 15 and the server device 9 described in the third embodiment.

  The medium driving device 1508 reads programs and data recorded in the portable storage medium 16 and writes data stored in the auxiliary storage device 1503 to the portable storage medium 16. For the medium driving device 1508, for example, a memory card reader / writer corresponding to one type or a plurality of types of standards can be used. When a memory card reader / writer is used as the medium driving device 1508, the portable storage medium 16 is a memory card (flash memory) conforming to a standard supported by the memory card reader / writer, for example, Secure Digital (SD) standard. ) Etc. can be used. As the portable recording medium 16, for example, a flash memory provided with a USB standard connector can be used. Further, when the computer 15 is equipped with an optical disk drive that can be used as the medium driving device 1508, various optical disks that can be recognized by the optical disk drive can be used as the portable recording medium 16. Examples of the optical disc that can be used as the portable recording medium 16 include a Compact Disc (CD), a Digital Versatile Disc (DVD), and a Blu-ray Disc (Blu-ray is a registered trademark). The portable recording medium 16 stores, for example, a voice processing program including the utterance content translation processing shown in FIG. 5 or 15 or a program in which the translation processing in step S9 in the flowchart shown in FIG. 5 or FIG. 15 is omitted. Is available. The portable recording medium 16 can be used as, for example, the voice holding unit 191, the calculation result holding unit 192, and the speaker information holding unit 193 shown in FIGS. 1 and 19. Furthermore, the portable recording medium 16 can be used for storing the translation dictionary 194 shown in FIG.

  When an operator inputs a voice processing start command to the computer 15 using the input device 1504 or the like, the processor 1501 reads and executes a voice processing program stored in a non-temporary recording medium such as the auxiliary storage device 1503. . In this processing, the processor 1501 functions (operates) as the first sound processing unit 110 and the second sound processing unit 120 in the sound processing apparatus 1 of FIG. 1 or FIG. In addition, the processor 901 acquires the bone conduction sound signal from the first microphone 2 and the air conduction sound signal from the second microphone 3 via the input / output interface 906. Further, while the processor 1501 performs the above-described sound processing, the RAM of the main storage device 1502, the auxiliary storage device 1503, and the like are stored in the sound data holding unit 191 and the calculation result holding unit in the sound processing device 1 of FIG. 192, which functions as a speaker information holding unit 193. When the computer 15 is operated as the speech processing apparatus 1 according to the first embodiment or the second embodiment, the RAM of the main storage device 1502, the auxiliary storage device 1503, and the like are used as storage units for storing the translation dictionary 194. It also has the function.

  Note that the computer 15 that operates as the voice processing device 1 does not need to include all the elements 1501 to 1508 illustrated in FIG. 19, and some elements may be omitted depending on applications and conditions. For example, the computer 15 operated as the voice processing apparatus 1 described in the first embodiment or the second embodiment may be one in which the communication control device 907 and the medium driving device 908 are omitted. Furthermore, the computer 15 operated as the voice processing device 1 may be a portable computer having a call function such as a smartphone.

The following additional notes are disclosed for each of the embodiments described above.
(Appendix 1)
Computer
When an utterance is detected according to a predetermined condition from a plurality of voice data collected simultaneously by a plurality of microphones, the utterance is traced back from the position where the utterance is detected in the voice data based on the predetermined condition. Set the start position of
Based on the characteristic difference from the start position of the utterance in each of the plurality of audio data to the position where the utterance is detected, the direction of the speaker viewed from the plurality of microphones is identified,
Based on the identified direction of the speaker, the speech section after the start position of the speech in any of the plurality of voice data is extracted.
A voice processing method characterized by executing processing.
(Appendix 2)
In the process of identifying the direction of the speaker, the computer is based on the characteristic difference for a section including a significant voice in a section from a start position of the utterance to a position where the utterance is detected in the voice data. Identifying the direction of the speaker,
The speech processing method according to supplementary note 1, wherein:
(Appendix 3)
In the process of extracting the utterance section of the voice data, the computer is configured to output the plurality of speech data based on the speaker position information in which the voice data is associated with the direction of the speaker viewed from the plurality of microphones. Extracting the utterance section in the audio data associated with the identified direction of the speaker from the audio data;
The speech processing method according to Supplementary Note 2, wherein
(Appendix 4)
In the process of setting the start position of the utterance, the computer divides the audio data into a plurality of frames, determines whether or not significant audio is included for each frame, and determines whether the significant in a plurality of consecutive frames Utterance detection is determined based on the number of frames that include a valid voice, and the frame that starts counting the number of frames that includes the significant voice is set as the start position of the utterance.
The speech processing method according to supplementary note 1, wherein:
(Appendix 5)
The computer performs a process of identifying the direction of the speaker each time the utterance is detected.
The speech processing method according to supplementary note 1, wherein:
(Appendix 6)
The computer acquires two pieces of sound data collected by a stereo microphone as the plurality of pieces of sound data;
The speech processing method according to supplementary note 1, wherein:
(Appendix 7)
After extracting the voice data, the computer further executes a process of translating the utterance content of the extracted voice data based on the utterance language associated with the direction of the speaker.
The speech processing method according to supplementary note 1, wherein:
(Appendix 8)
After extracting the audio data, the computer
Sending the translation request including the extracted voice data, the language of the translation source, and the language of the translation destination to a server device capable of executing the translation process of the voice data,
A process of receiving and outputting a result of translation processing on the audio data from the server device;
The speech processing method according to supplementary note 1, wherein:
(Appendix 9)
When an utterance is detected according to a predetermined condition from a plurality of voice data collected simultaneously by a plurality of microphones, the utterance is traced back from the position where the utterance is detected in the voice data based on the predetermined condition. An utterance detection unit for setting the start position of
A direction identifying unit for identifying a direction of a speaker viewed from the plurality of microphones based on a characteristic difference from a start position of the utterance to a position where the utterance is detected in each of the plurality of voice data;
A voice data extraction unit that extracts a utterance section after the start position of the utterance in any of the plurality of voice data based on the direction of the identified speaker;
An audio processing apparatus comprising:
(Appendix 10)
When an utterance is detected according to a predetermined condition from a plurality of pieces of voice data collected simultaneously by a plurality of microphones, based on the predetermined condition, the position of the utterance in the voice data is traced back to the past. Set the utterance start position,
Based on the characteristic difference from the start position of the utterance in each of the plurality of audio data to the position where the utterance is detected, the direction of the speaker viewed from the plurality of microphones is identified,
Based on the identified direction of the speaker, the speech section after the start position of the speech in any of the plurality of voice data is extracted.
A voice processing program that causes a computer to execute processing.

DESCRIPTION OF SYMBOLS 1 Audio | voice processing apparatus 2 Stereo microphone 3 Display apparatus 4A, 4B Speaker 9 Server apparatus 15 Computer 16 Portable recording medium 110 1st audio | voice processing part 111 Audio | voice data acquisition part 112 Holding data control part 120 2nd audio | voice processing part 121 Speech detection unit 122 Direction identification unit 122A Frame power calculation unit 122B Average power calculation unit 122C Sound pressure difference calculation unit 122D Speaker direction determination unit 123 Language switching unit 124 Speech data extraction unit 125, 910 Translation processing unit 126 Output unit 130 Speaker information Setting unit 140, 920 Communication unit 191 Audio data holding unit 192 Calculation result holding unit 193 Speaker information holding unit 194, 991 Translation dictionary 201, 202 Microphone 1501 Processor 1502 Main storage device 1503 Auxiliary storage device 1504 Input device 1505 Output device 1506 I / O interface Over scan 1507 communication control unit 1508 medium driving device 1510 bus

Claims (8)

Computer
When an utterance is detected according to a predetermined condition from a plurality of voice data collected simultaneously by a plurality of microphones, the utterance is traced back from the position where the utterance is detected in the voice data based on the predetermined condition. Set the start position of
Based on the characteristic difference from the start position of the utterance in each of the plurality of audio data to the position where the utterance is detected, the direction of the speaker viewed from the plurality of microphones is identified,
Based on the identified direction of the speaker, the speech section after the start position of the speech in any of the plurality of voice data is extracted.
A voice processing method characterized by executing processing. In the process of identifying the direction of the speaker, the computer is based on the characteristic difference for a section including a significant voice in a section from a start position of the utterance to a position where the utterance is detected in the voice data. Identifying the direction of the speaker,
The voice processing method according to claim 1. In the process of extracting the utterance section of the voice data, the computer is configured to output the plurality of speech data based on the speaker position information in which the voice data is associated with the direction of the speaker viewed from the plurality of microphones. Extracting the utterance section in the audio data associated with the identified direction of the speaker from the audio data;
The voice processing method according to claim 2. In the process of setting the start position of the utterance, the computer divides the audio data into a plurality of frames, determines whether or not significant audio is included for each frame, and determines whether the significant in a plurality of consecutive frames Utterance detection is determined based on the number of frames that include a valid voice, and the frame that starts counting the number of frames that includes the significant voice is set as the start position of the utterance.
The voice processing method according to claim 1. The computer performs a process of identifying the direction of the speaker each time the utterance is detected.
The voice processing method according to claim 1. After extracting the voice data, the computer further executes a process of translating the utterance content of the extracted voice data based on the utterance language associated with the direction of the speaker.
The voice processing method according to claim 1. When an utterance is detected according to a predetermined condition from a plurality of voice data collected simultaneously by a plurality of microphones, the utterance is traced back from the position where the utterance is detected in the voice data based on the predetermined condition. An utterance detection unit for setting the start position of
A direction identifying unit for identifying a direction of a speaker viewed from the plurality of microphones based on a characteristic difference from a start position of the utterance to a position where the utterance is detected in each of the plurality of voice data;
A voice data extraction unit that extracts a utterance section after the start position of the utterance in any of the plurality of voice data based on the direction of the identified speaker;
An audio processing apparatus comprising: When an utterance is detected according to a predetermined condition from a plurality of voice data collected simultaneously by a plurality of microphones, the utterance is traced back from the position where the utterance is detected in the voice data based on the predetermined condition. Set the start position of
Based on the characteristic difference from the start position of the utterance in each of the plurality of audio data to the position where the utterance is detected, the direction of the speaker viewed from the plurality of microphones is identified,
Based on the identified direction of the speaker, the speech section after the start position of the speech in any of the plurality of voice data is extracted.
A voice processing program that causes a computer to execute processing.