JP2010086074A - Speech processing apparatus, speech processing method, and speech processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently create document data with speech data added thereto. <P>SOLUTION: A speech document correspondence unit 22 of a speech processing apparatus 1 calculates the likelihood of each word in a document included in document data 20 from a speech frame sequence 13 to be created in order to detect a first time period of the speech frame sequence 13 during which words in the document are spoken and a second time period of the speech frame sequence 13 during which contents other than the words in the document are spoken, extracts, as a symbol, the speech frame sequence 13 of the second time period, and decides, as an additional position of the extracted symbol, an adjacent position of a position of a word in the document spoken during the first time period which is temporally close with respect to the second time period during which the extracted symbol is spoken. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a voice processing device, a voice processing method, and a voice processing program.

A document processing apparatus that attaches input audio data to document data has been proposed (Patent Document 1). In this document processing apparatus, when a user designates an attachment position in document data, audio data is attached to the attachment position. Note that examples of the audio data attachment format include a format in which an icon indicating that audio is attached and a format in which the audio data is attached as a character string converted into a text by voice recognition.
Japanese Patent Laid-Open No. 11-242669

  In the conventional document processing apparatus such as Patent Document 1, it is necessary to manually input the attachment position of the audio data one by one, so that the cost of creating the document data with the audio data is high.

  Furthermore, recording of audio data is often performed continuously for a long period of time at a conference or the like. At that time, the audio data file is not a file for each attachment position, but is a file for the entire meeting. Therefore, it is necessary to cut out a file for each of a plurality of attachment positions from the file of one entire meeting, and the cost for creating document data increases.

  Accordingly, the main object of the present invention is to solve the above-described problems and efficiently create document data with audio data.

In order to solve the above problems, the present invention is an audio processing apparatus for adding sampled waveform data to document data as a symbol,
The speech processing apparatus includes a speech frame sequence detection unit, a speech document correspondence unit, and a symbol addition unit,
The voice frame sequence detection unit,
Read the waveform data from the storage unit, create a voice frame sequence by detecting a voice section based on the volume of the waveform data,
The voice document correspondence unit
By calculating the likelihood for each word in the document included in the document data from the generated voice frame sequence, the first time zone of the voice frame sequence in which the word in the document is spoken, and a word other than the word in the document Detecting a second time zone of the audio frame sequence in which the content is spoken;
The speech frame sequence in the second time zone is extracted as a symbol, and the word in the document uttered in the first time zone that is closer in time to the second time zone in which the extracted symbol is uttered. The position near the position is determined as an additional position of the extracted symbol,
The symbol adding unit is
Based on the position of the word in the document read from the storage unit and the addition position of the symbol determined by the voice document corresponding unit, the symbol is arranged in the document data, so that the extracted symbol is the document. It is added to data.
Other means will be described later.

  According to the present invention, document data with audio data can be created efficiently.

  Hereinafter, first and second embodiments of the present invention will be described. The common points of the first and second embodiments are that when the explainer explains along the document, the content of the remarks not described in the document is recorded with the microphone, and the recorded content is added to the document as supplementary data. It is a point to do. By adding supplementary data, it can be used for the following applications, for example.

  For example, the explainer explains to the consumer the explanation of the product, important matters of the contract, etc. in the business. Explainers often repeat the same explanations for many customers, so by adding the contents of the explanation to the document as supplementary data, it is possible to easily grasp the places where supplementary explanations are necessary and improve the efficiency of explanation work. can do.

  In addition, the presenter gives explanations of the type for which legal consent is required in the company, at the educational site, or at the support center. It is effective to record audio and other media (videos, memos, etc.) in order to keep the contents of the conversation conducted during the explanation and the consent of the explained partner as evidence.

Hereinafter, a first embodiment of the present invention will be described. In the first embodiment, a speaker microphone and a speaker microphone are prepared, and the sound source of the speaker is specified from each microphone. Therefore, the first embodiment can be applied to a description on a telephone such as a call center.
Specifically, when an agent speaks an inappropriate word at the call center, it detects it in real time and notifies the supervisor. It can be applied to systems that display on the screen.

FIG. 1 is a configuration diagram illustrating a voice processing device. The sound processing apparatus 1 is configured as a general computer having a CPU (Central Processing Unit) 85, a memory 86, and an interface.
The CPU 85 executes arithmetic processing and control processing of each component of the sound processing device 1.
The interface of the sound processing device 1 connects the speaker microphone 80, the listener microphone 81, the display 91, the mouse 92, and the speaker 93 to the sound processing device 1.

  The speech processing apparatus 1 is assumed to have a mechanism in which an arithmetic device performs processing based on a software program stored in a storage device, as in a general computer, but the present invention does not depend on such a configuration, For example, the present invention can be applied to an architecture that implements functions equivalent to software programs in hardware.

The explainer microphone 80 records (samples) the explainer's voice and outputs the result to the waveform data 10.
The listener's microphone 81 records the voice of the listener (explained person) and outputs the result to the waveform data 10.
The display 91 is a display device that displays data viewed by the user, such as the document data 20 and the document data 43 with symbols.
The mouse 92 is an input device that accepts an operation of a mouse pointer displayed on the display 91.
The speaker 93 is a device that outputs the waveform data 10 as an audio signal.

The memory 86 stores a program for realizing each processing unit when executed by the CPU 85 and each data to be processed by each processing unit.
The audio processing device 1 includes an audio frame sequence detection unit 12, an audio document correspondence unit 22, a symbol creation unit 40, a symbol addition unit 42, and a symbol reproduction unit 44 as processing units.
The speech processing apparatus 1 includes, as data, waveform data 10, waveform frames 11, speech frame sequence 13, document data 20, in-document word phonological model 21, in-document word likelihood map 22a, speech document correspondence data 22b, general word dictionary 31, general phoneme model 32, symbol data 41, and document data with symbol 43.
Hereinafter, details of each component of the voice processing device 1 will be described.

  The in-document word phoneme model 21 is created in advance by serially connecting general phoneme models 32 corresponding to each phoneme constituting the in-document word for each in-document word appearing in the document data 20.

The general word dictionary 31 is not limited to the words in the document of the document data 20, but includes general words that can be spoken.
The general phoneme model 32 is created by learning each word in the general word dictionary 31 from a large number of waveforms using an HMM (Hidden Markov Model), and is used to determine the output probability of the HMM. As a general method used in word speech recognition, the output probability of the HMM is calculated by calculating the distance between MFCC (Mel Frequency Cepstrum Coefficient) feature vectors of the general phoneme model 32 and the speech frame sequence 13. Can be sought.

  FIG. 2 is an explanatory diagram showing acoustic signal data. The waveform data 10, the waveform frame 11, and the voice frame sequence 13 as acoustic signal data will be described. In FIG. 2, in order to show the time-series waveform data 10, a graph is shown in which the horizontal axis represents time and the vertical axis represents volume.

  The waveform data 10 is time-series data recorded as a digital signal by A / D converting acoustic signal data input via a microphone (explainer microphone 80, listener microphone 81) for each sound source. . The waveform data 10 includes a time zone in which an explainer or the like is speaking, and a time zone in which there is no speech due to silence or the like. The sampling environment of the waveform data 10 is, for example, a sampling frequency of 10 [kHz], a quantization of 8 [bit], and an encoding method of linear PCM (pulse code modulation).

  The waveform frame 11 is data obtained by dividing the waveform data 10 by a unit length (fixed length) time zone. The unit length of the waveform frame 11 is an arbitrary length suitable for use in speech analysis. For example, the unit length of the waveform frame 11 is set to 500 samples (20 [ms]). Since the waveform frame 11 only divides the time zone, like the waveform data 10, there is a time zone in which a speech is made and a time zone in which there is no speech.

The audio frame sequence 13 is data in which a sequence of one or more continuous waveform frames 11 is collected. As a result of signal processing of the waveform frame 11, the audio frame sequence 13 is configured as data in a time zone in which speech is continued from the waveform frame 11 at the start of speech to the waveform frame 11 at the end of speech.
The signal processing for the waveform frame 11 is, for example, a process for comparing the volume and the threshold value for each waveform frame 11, and in FIG. 2, the waveform frame 11 that has exceeded the start threshold value α has fallen below the end threshold value β. Up to the waveform frame 11 is extracted as one audio frame sequence 13. As described above, the method using the two threshold values (start threshold value α and end threshold value β) together is called hysteresis, and the extracted audio frame sequence 13 is compared with a method for determining using only one threshold value. This has the effect of preventing chopping.

  FIG. 3A is an explanatory diagram showing the content of a statement in the waveform data 10 as text. The horizontal direction is the time axis, and the vertical direction is the time and each sound source (explainer, listener). Three rows in the vertical direction are set as one set, and three sets are folded back at every time “1000 [ms]”. The time zone in which the text is described is the time zone of the audio frame sequence 13.

  FIG. 3B is an explanatory diagram in which the recognition result of the word in the document indicated by the voice document correspondence data 22b (Table 3, details will be described later) is overwritten on the waveform data 10 shown in FIG. . For example, referring to the voice document correspondence data 22b, the word “notes” in the document “# 1” is spoken during the voice time “100 to 300”, and therefore “#” in the “notes” column of the explainer column. 1 "is assigned. Referring to FIG. 3 (b), supplementary explanations not described in the document data 20 ("first" at time = 1000, etc.), consent by the listener ("Okay" at time = 2000, Etc.).

FIG. 4A is a screen diagram illustrating a display example of the document data 20 (Table 1, details will be described later) output from the speech processing apparatus 1 to the display 91.
The display position (layout) of characters in the document data 20 such as “notes” is determined based on the display position (range from the upper left coordinate to the lower right coordinate) stored in the document data 20. Although described as the display position, in detail, since it indicates the position of the word in the document in the document data 20, it is the print position in the printing process and the storage position in the file saving process.

FIG. 4B is a screen diagram showing a display example of the document data 43 with symbols, which is a result of adding the symbol data 41 (Table 5, details will be described later) to FIG. The display positions of the balloons 711 to 719 are determined based on the “symbol display position” of the symbol data 41.
Gray balloons 713 and 716 indicate the content of the listener's speech. The display character strings (Yes, I understand) of these balloons 713 and 716 are “display character strings” of the symbol data 41.
White balloons 711, 712, 714, 715, 717 to 719 indicate the content of the commentary of the presenter. Since the display character strings of these balloons cannot be displayed completely due to the empty space in the screen, they are indicated by a short form such as “..”. The size of the speech bubble is different from the time length of each symbol indicated by the speech bubble. The longer the speech time, the larger the speech bubble is displayed. In this manner, by reflecting the time length for each symbol indicated by the balloon in the screen display content of the symbol, the user can intuitively grasp the time length for each symbol.
In addition to the size of the above-described symbol balloon, for example, the symbol color may be changed based on the time length, or the symbol time length (number (Notation) may be included in the symbol display as a character string. Furthermore, the process for reflecting the above-described time length in the display content may be only one type, or a plurality of types may be combined.
Here, in FIG. 4B, the difference in color between the balloons indicates the difference in sound source, and the “display color” of the symbol data 41 determined based on the “sound source” of the symbol data 41 is the same. Reflected.

Table 1 shows the document data 20. In the document data 20, for each word in the document, its ID “#”, its importance, and display position information (upper left coordinates, lower right coordinates) are associated.
The word in the document is a unit delimited at a place where there is a possibility of being delimited when the explainer speaks.
The importance may be given in advance, or may be determined from an important word dictionary, part of speech, or the like.
The display position information is information serving as a layout for displaying document data (see FIG. 4).

  Table 2 is an explanatory view showing the in-document word likelihood map 22a. Table 2 shows an extracted portion of the word “code” and “do not pull” in the document from the word likelihood map 22a in the document created when 2250 waveform frames 11 are input. . The word likelihood map 22a in the document includes a waveform frame number, a “score” (maximum likelihood) up to the waveform frame number for each word in the document, and a waveform frame number that outputs the “score”. "Start frame number" is recorded.

The fifteen waveform frames 11 corresponding to the waveform frame numbers “1 to 15” include the following two waveform frames 11 of words in the document.
A waveform frame 11 of the word “code” in the document corresponding to the waveform frame number “2-3”. This is because the start frame number of the waveform frame number “3”, which is the highest score “7” of the word “code” in the document, is “2”.
A waveform frame 11 of the word “do not pull” in the document corresponding to the waveform frame number “8 to 11”. This is because the start frame number of the waveform frame number “11” that is the highest score “9” of the word “do not pull” in the document is “8”.

On the other hand, the following waveform frames 11 in which the above-mentioned words in the document are not spoken are respectively symbol data 41.
-Waveform frame 11 with waveform frame number "1"
-Waveform frame 11 of waveform frame number "4-7"
-Waveform frame 11 of waveform frame number "12-15"

  Table 3 shows the voice document correspondence data 22b of the presenter. The voice document correspondence data 22b is obtained by associating the waveform data 10 with the document data 20 based on the speech time of words in the document. The voice document correspondence data 22b indicates, for each column of the waveform frame 11, the ID “#”, the text content of the word in the document (“speech outside document” if it is not a word in the document), and the time zone in which the speech was made. The speech time (start time to end time) is associated with the display coordinates of the words in the document on the screen.

  Table 4 shows the audio document correspondence data 22b of the listener. Since all the contents of the listener's utterances are out-of-document audio, the “in-document word” column in Table 3 is replaced with the “out-of-document audio” column. The other columns in Table 4 have the same format as Table 3.

  Table 5 is an explanatory diagram showing the symbol data 41. The symbol data 41 includes a “symbol code (symbol code in FIG. 4B)” indicating a symbol ID, symbol information (sound source, display color, display character string), and symbol display position. Configured in association.

  FIG. 5 is a flowchart showing the operation of the audio frame sequence detector 12 (FIG. 1). As shown in FIG. 2, the audio frame sequence detector 12 extracts the audio frame sequence 13 from the waveform frame 11. Note that the extraction process of the audio frame sequence 13 is a process generally called audio section detection. The method described with reference to FIG. 5 is an example thereof, and can be replaced with an arbitrary voice segment detection process.

5 is executed independently for each of the waveform data 10 from the explainer microphone 80 and the waveform data 10 from the listener microphone 81.
First, since the audio frame sequence 13 generated from the waveform data 10 of the explainer microphone 80 includes the utterance content of the word in the document, the audio document correspondence unit 22 creates the in-document word likelihood map 22a. Input data.
On the other hand, since the audio frame sequence 13 generated from the waveform data 10 from the listener's microphone 81 does not include the content of words in the document, it is not used to create the word likelihood map 22a in the document.

  In S101, the waveform data 10 continuously sampled from the microphones (explainer microphone 80, listener microphone 81) is divided into unit lengths in order of time series and read as a waveform frame 11. Hereinafter, the loop from S101 to S110 is a loop for selecting the read waveform frame 11.

As S102, the power p indicating the volume of the waveform frame 11 being selected in S101 is calculated. The power p is calculated as the square sum of the amplitudes of all the samples in the waveform frame 11.
In S103, if the power p is larger than the start threshold value α (see FIG. 2) (S103, Yes), the recording flag “F” is set to 1 (S103, Yes) to set the selected waveform frame 11 as the start frame of the audio frame sequence 13. (Recording) is substituted (S104).
As S105, when the recording flag “F” is “1” (S105, Yes), the selected waveform frame 11 is recorded in the audio buffer (S106).

In S107, if the power p is smaller than the end threshold β (see FIG. 2) (S107, Yes), the recorded audio frame sequence 13 is recorded with the waveform frame 11 recorded in the previous S106 as the end frame of the audio frame sequence 13. In addition to outputting as output data of the audio frame sequence detector 12, the audio buffer is cleared to record the next audio frame sequence 13 (S108). Then, 0 (not recording) is substituted for the recording flag “F” (S109).
In S110, the loop is terminated.

  The voice document correspondence unit 22 associates the voice frame sequence 13 output from the voice frame sequence detection unit 12 with the document data 20. When the sound source of the audio frame sequence 13 is the explainer, the audio document correspondence unit 22 performs the operation shown in FIG. 6, and based on the in-document word likelihood map 22a shown in Table 2, the table shown in Table 3 follows. The voice document correspondence data 22b is created.

  FIG. 6 is a flowchart showing the operation of the voice document correspondence unit 22 when the sound source is an explainer. This flowchart is executed every time the voice document correspondence unit 22 inputs the voice frame sequence 13 output from the voice frame sequence detection unit 12. That is, by executing the process of FIG. 6 once, one audio frame sequence 13 is processed.

First, the loop process of S201 to S203 is a process in which the speech document correspondence unit 22 generates the in-document word likelihood map 22a as the probability likelihood when the speech frame sequence 13 is input to the in-document word phoneme model 21. is there.
In S201, a loop for selecting words in the document data 20 one by one in order is started.
In S202, the word likelihood (score) of the word in the document selected in S201 is calculated for each waveform frame 11 constituting the speech frame sequence 13, and written to the word likelihood map 22a in the document.
In S203, the loop is terminated. For example, the “code” column in Table 2 is generated in one loop, and the “do not pull” column in Table 2 is generated in the next loop.

Next, the processing of S211 to S213 is processing for generating voice document correspondence data 22b with reference to the generated intra-document word likelihood map 22a.
In S211, the optimum path of the waveform frame number f is searched for each word w in the document in the word likelihood map 22a in the document.
Specifically, when the word score of the word w in the document in the waveform frame number f is score (w, f) and the start position is begin (w, f), the optimal path ({wi, fi}) is It is calculated | required by (Formula 1).

In the example of Table 2, {w1 = 5, f1 = 3} in the word “code” column in the document and {w2 = 6, f2 = 11} in the word “do not pull” column in the document are optimal paths. It becomes.
Table 2 shows only some of the words (w1 = 5, w2 = 6), but the likelihood scores of the other words are low because the waveform frame does not match the word phoneme model and the likelihood score is low. The best path is not selected.

  Furthermore, the “importance” of the document data 20 may be taken into consideration for the formula for obtaining the optimum path. When the “importance” of the word w in the document is P (w), the optimum path ({wi, fi}) is obtained by the following (Formula 2).

When the optimum path for one word w in the document has already been found once, the search for the optimum path for the word w in the document may be omitted. This omitted case is based on the premise that a word in a document is spoken only once in a presentation or the like, and the amount of calculation can be reduced by omitting a path search.
On the other hand, when a word in a document is replied a plurality of times in a question-and-answer session or the like, it is better not to omit the path search because the above-mentioned assumption is not satisfied.

  In S212, the waveform frame sequence is divided into in-document words and out-of-document speech according to the optimum path obtained in S211. The word in the document is a section (begin (wi, fi), fi) of the word wi selected by the optimum path. The voice outside the document is a section of a continuous waveform frame sequence other than the words in the document.

In S213, the word in the document obtained in S212 and the voice outside the document are written in the voice document correspondence data 22b (Table 3).
In the “word in document” column of the voice document correspondence data 22b, text indicating the word in the document obtained in S212 (obtained from the document data 20) or a flag indicating voice outside the document is described.
In the “voice time” column of the voice document correspondence data 22b, the waveform frame 11 constituting the word in the document and the voice outside the document obtained in S212 is counted from the start time of the first waveform frame 11 to the end of the last waveform frame 11. The time zone up to the time is described.
In the “display coordinates” column of the voice document correspondence data 22b, the position (the position of the tip portion of the balloon in the balloon icon) where the voice outside the document obtained in S212 is displayed on the display 91 based on the document data 20 is described.

The “display coordinates” of the voice outside the document can be obtained from the following method based on, for example, the time close to the speech time zone between the voice outside the document and the word in the document.
First, when an out-of-document voice (# 1) is uttered and an in-document word ("Notes" in # 2) is uttered, the in-document word (# 1) is referred to the document data 20 (Table 1). "Precautions") display position (upper left coordinates, lower right coordinates). Then, the coordinates in the vicinity (slightly left) of the upper left coordinates of the words in the document are set as “display coordinates” of the voice outside the document.
Next, when an out-of-document voice (# 3) is uttered after an in-document word (# 2 “notes”) is uttered, the in-document word (#) is referred to the document data 20 (Table 1). 1 ("Notes") display position (upper left coordinates, lower right coordinates) is acquired. Then, the coordinates in the vicinity (slightly right) of the upper right coordinates of the words in the document are set as “display coordinates” of the voice outside the document.

The processing when the sound source of the audio frame sequence 13 is the explainer has been described above with reference to FIG. On the other hand, when the sound source of the voice frame sequence 13 is a listener, the voice document correspondence unit 22 creates the voice document correspondence data 22b shown in Table 4 based on the voice document correspondence data 22b shown in Table 3.
Specifically, the speech document correspondence unit 22 (symbol text creation unit) performs word speech recognition for each out-of-document speech using the general phoneme model 32 and the general word dictionary 31.
Specifically, the word speech recognition is based on the scores S (1), S (2),..., S (J) of the words W1, W2,. Find using. If the maximum score value max (S (j))> γ (γ is a predetermined threshold value), the recognition result Wj | j = argmax (S (J)) is set as the display character string. Otherwise, a symbol such as “?” Indicating a recognition failure is used as a display character string.

Then, the voice document correspondence unit 22 refers to the presenter's voice document correspondence data 22b (Table 3) and determines the “display coordinates” of the listener's voice document correspondence data 22b (Table 4).
For example, the out-of-document voice “Yes” (voice time = 700 to 800) in Table 4 is remarked after the voice time “100 to 300” of the word “Notes” in Table 3. Therefore, the display coordinate of the voice outside the document “Yes” is the same position as the display position “(140,100)” of the voice outside the document (voice time = 300 to 600) in Table 3 immediately after the word “Notes” in the document. To do.

  Returning to FIG. 1, the symbol creation unit 40 creates symbol data 41 (Table 5) from the voice document correspondence data 22b (Tables 3 and 4).

  The symbol creation unit 40 extracts, as a symbol, an out-of-document voice excluding in-document words from the presenter's voice document correspondence data 22b (Table 3). For example, the symbol creation unit 40 uses the out-of-document voice (# 1) with the voice time “0 to 100 [ms]” in Table 3 as the symbol with the symbol code “711”, and displays the “display coordinates” in Table 3. 5 is substituted for “symbol display position”.

  The symbol creation unit 40 extracts all out-of-document sounds as symbols from the audio document correspondence data 22b (Table 4) of the listener. For example, the symbol creating unit 40 uses the out-of-document voice (# 1, yes) of the voice time “700 to 800 [ms]” in Table 4 as the symbol with the symbol code “713”, and the “display coordinates” in Table 4 Is substituted for “symbol display position” in Table 5, and the text “sound outside document” in Table 4 is substituted for “display character string” of the symbol in Table 5.

As for the “display character string” of the symbol data 41, basically, the voice content converted into text as a result of the voice recognition process is described as it is. However, when a speech recognition result is not obtained, such as when a word outside the dictionary is spoken, if the length of the out-of-document speech is equal to or less than a predetermined length according to the time length of the out-of-document speech, “-(none) “, Or a character string“...
The “display color” of the symbol data 41 is “white” if the “sound source” is the explainer, and “gray” if the “sound source” is the listener.

  The symbol reproducing unit 44 outputs the audio signal of the designated symbol data 41 from the speaker 93. The symbol data 41 can be specified by using, for example, the symbol data 41 of the document data with symbol 43 displayed on the display 91 using the mouse 92 (balloons 711 to 719 in FIG. 4B). Click to execute.

  Then, when the symbol data 41 is designated, the symbol reproduction unit 44 refers to the voice document correspondence data 22b that is the creation source of the symbol data 41, acquires the “voice time” of the voice document correspondence data 22b, and The waveform data 10 for “voice time” is read from the memory 86 and output from the speaker 93. Thereby, the user can hear the sound corresponding to the designated display character string.

The first embodiment has been described above. Thereby, the user can obtain a question or a response from the audience by referring to the symbol data 41 (balloon icon) added to the displayed symbol-added document data 43, where the explanation is supplemented by the explanation. As a result, the contents of the remarks can be understood immediately.
Furthermore, instead of displaying and explaining the document data 20 on the screen, the explainer displays the symbol-added document data 43 and explains the contents so that the explanation can proceed efficiently while confirming the past explanation contents. Can do.

  A second embodiment of the present invention will be described. In the second embodiment, sound is recorded via a microphone array that can specify the sound source of sound to be recorded. This eliminates the need to prepare a microphone for each speaker, thus facilitating introduction. For example, the second embodiment is implemented in a facing environment such as a conference room.

FIG. 7 is a configuration diagram illustrating the speech processing apparatus according to the second embodiment. Compared with FIG. 1, each microphone (explainer microphone 80, listener microphone 81) in FIG. 1 is replaced with a microphone array 82 including a plurality of microphone elements 82a to 82c in FIG.
The microphone array 82 is installed in the vicinity of the explanation scene. This eliminates the need to use a proximity microphone such as a headset microphone for each explainer, so that the explanation can be made smoothly without being conscious of the installation of the microphone. The sound source position is represented by the sound source direction θ of the azimuth angle viewed from the microphone array 82.

  In addition to the components shown in FIG. 1, a sound source identification unit 14 is added to the voice processing apparatus 1 shown in FIG. The sound source identification unit 14 detects the sound source position by examining the time difference at which the waveform data 10 from the sound source reaches the microphone elements 82a to 82c. Further, the sound source identification unit 14 applies a spatial filter corresponding to the sound source position, emphasizes and extracts the sound from the sound source position, and outputs it to the waveform data 10.

FIG. 8 is a flowchart showing the operation of the audio frame sequence detection unit 12 of the second embodiment. Compared with the flowchart of FIG. 5, the following points are changed.
A loop (S101a to S110a) for each sound source direction θ is added.
The power p of the waveform frame 11 is changed to the power p (θ) for each sound source direction θ (S102b, S103b, S107b). The sound source direction θ is assumed to be an azimuth angle of 360 degrees with respect to the microphone array at intervals of 5 degrees (θ = 0, 5,..., 355). p (θ) is the complex conjugate transposed vector x ^* of the Fourier transform x = {xi} of the input waveform frame to the microphone element {i = 0, 1, 2, 3} and the steering vector a corresponding to the sound source direction θ. From (θ), p (θ) = a (θ) · x ^* is obtained.
The recording flag “F” is changed to the recording flag “F (θ)” for each sound source direction θ (S104b, S105b, S108b).
In S109b, in addition to the audio frame sequence 13 output in S109, the sound source direction θ selected in S101a is also output. Here, by outputting x (θ) = a (θ) x in which the sound source direction θ is emphasized by the delay sum beamforming, it is possible to reduce the influence of sound other than the sound source direction θ recorded by the microphone array 82.

FIG. 9 is a flowchart showing the operation of the voice document handling unit 22 of the second embodiment. Compared with the flowchart of FIG. 6, the following points are changed. With this change, the sound source identification unit 14 can detect the direction of the presenter even for the input sound obtained by the microphone array 82, and the contents of the explanation can be visualized.
A document word likelihood map 22a is generated and referenced for each sound source direction θ (S202b, S211b).
In the flowchart of FIG. 6, it is assumed that the audio frame sequence 13 of the explainer is input. However, in the flowchart of FIG. 9, whether or not an explainer exists in the sound source direction θ is not specified at the start of execution. Therefore, the sound source identification unit 14 determines whether there is an explainer in the sound source direction θ by executing processing (S211c, S211d) of the newly added variable S (θ).

The variable S (θ) indicating the character of the explainer is introduced in order to determine whether or not the explainer exists in the sound source direction θ based on whether the explanatory text is included in the content of the statement or whether it has been included in the past. Variable.
The sound source identification unit 14 calculates the sum SP (θ) of the scores in the in-document speech section of the optimum path by the following (Equation 3).

Using this SP (θ), S (θ) ← γS (θ) + (1-γ) SP (θ) is updated (S211c). Here, γ represents a forgetting factor, for example, γ = 0.9.
The sound source identification unit 14 determines whether or not S (θ) updated in S211c exceeds a preset threshold value ν. When the threshold value ν is exceeded (S211d, Yes), it is determined that there is an explainer in the sound source direction θ, and S212 is executed. On the other hand, when the threshold value ν is not exceeded (S211d, No), it is determined that there is no presenter in the sound source direction θ. When there is no presenter in the sound source direction θ, the speech document correspondence unit 22 regards the entire speech frame sequence 13 input at the start of FIG. 9 as one out-of-document speech (S212a), and advances the processing to S213. .

  If the sound source direction θ of the presenter is given in advance, the processing of S211c of the sound source identification unit 14 may be omitted and the determination formula of S211d may be replaced with “the presenter exists in the sound source direction θ”. Good.

  Returning to FIG. 7, the operation of the symbol creation unit 40 is the same as that of the first embodiment of FIG. Here, the display color may be changed for each sound source direction θ. Thereby, even if there are a plurality of explainers and listeners, it is possible to indicate on the screen which statement has been made by the same person.

According to the first and second embodiments described above, when the explainer explains along the document, it is possible to add the remarks content not described in the document as supplementary data to the document.
In this way, when recording the audio at the time of explanation and leaving it as proof of explanation, the instructor has access to information during the explanation about which of the document material was explained or how the listener responded to it. can do.
Therefore, it is possible to eliminate omissions in explanation and omission of consent by efficiently referring to the contents of statements that are not described in the document. In other words, in a situation where the explainer explains to another person based on the document material, it is easy to understand and carefully selected information can be provided as a symbol to the explainer.

Each embodiment of the present invention is effective in applications such as structuring and visualization of meetings and the like.
As a comparative example for visualizing the content of a plurality of speakers, there is a method in which a speech for each speaker is detected and displayed as a time series graph.
As comparative examples using document materials, there are a case where a slide displayed during a meeting is displayed in association with another, and a case where the slide is used for adaptation of a language model used in speech recognition.
In each embodiment of the present invention, in the application of visualizing the scene to be explained based on the document material, the content of each speaker's speech is more visually (balloon icon on the screen) more intuitively than the comparative example described above. In addition, it is possible to provide a usage method that is easy to understand auditorially (sound reproduction by clicking a speech bubble icon).

It is a block diagram which shows the audio | voice processing apparatus regarding 1st Embodiment of this invention. It is explanatory drawing which shows the acoustic signal data regarding 1st Embodiment of this invention. It is explanatory drawing which shows the statement content of the waveform data regarding 1st Embodiment of this invention. It is a screen figure which shows the example of a display of the document data output on the display regarding 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the audio | voice frame sequence detection part regarding 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the audio | voice document corresponding | compatible part regarding 1st Embodiment of this invention. It is a block diagram which shows the audio | voice processing apparatus regarding 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the audio | voice frame sequence detection part regarding 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the audio | voice document corresponding | compatible part regarding 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Speech processing apparatus 10 Waveform data 11 Waveform frame 12 Speech frame sequence detection part 13 Speech frame sequence 14 Sound source identification part 20 Document data 21 In-document word phoneme model 22 Spoken document correspondence part 22a In-document word likelihood map 22b Spoken document correspondence data 31 General Word Dictionary 32 General Phoneme Model 40 Symbol Creation Unit 41 Symbol Data 42 Symbol Addition Unit 43 Document Data with Symbol 44 Symbol Reproduction Unit 80 Microphone for Explainer 81 Microphone for Audience 82 Microphone Array 82a-82c Microphone Element 85 CPU
86 Memory 91 Display 92 Mouse 93 Speaker

Claims (14)

An audio processing apparatus for adding sampled waveform data to document data as symbols,
The audio processing device includes an audio frame sequence detection unit, an audio document correspondence unit, and a symbol addition unit,
The voice frame sequence detector
Read the waveform data from the storage unit, create a voice frame sequence by detecting the voice section of the waveform data,
The voice document support unit
By calculating the likelihood for each word in the document included in the document data from the generated voice frame sequence, the first time zone of the voice frame sequence in which the word in the document is spoken, and a word other than the word in the document Detecting a second time zone of the audio frame sequence in which the content is spoken;
The speech frame sequence in the second time zone is extracted as a symbol, and the word in the document uttered in the first time zone that is closer in time to the second time zone in which the extracted symbol is uttered. The position near the position is determined as an additional position of the extracted symbol,
The symbol adding unit includes:
Based on the position of the word in the document read from the storage unit and the addition position of the symbol determined by the voice document corresponding unit, the symbol is arranged in the document data, so that the extracted symbol is the document. A voice processing device characterized by being added to data. When calculating the likelihood for each word in the document, the voice document correspondence unit reads the importance for each word in the document from the storage unit, and the word in the document is remarked based on the likelihood and the importance. The audio processing device according to claim 1, wherein the first time zone of the audio frame sequence and the second time zone of the audio frame sequence in which contents other than words in the document are spoken are detected. . The audio processing apparatus further includes a symbol reproduction unit,
When the symbol displayed on the screen of the display device is selected through the input device, the symbol playback unit controls to play back the audio frame sequence of the selected symbol through a speaker. The speech processing apparatus according to claim 1 or 2. The speech processing apparatus further includes a symbol text creation unit,
The symbol text creation unit reads a phonological model indicating a phoneme of a word in a word dictionary from a storage unit, and inputs the speech frame sequence of the symbol to the phonological model, thereby the word dictionary included in the speech frame sequence. The speech processing apparatus according to any one of claims 1 to 3, wherein a word is extracted, the utterance content of a symbol is converted into text, and the text is added to the document data. The voice processing according to any one of claims 1 to 4, wherein the symbol adding unit reflects a time length of a symbol time zone for a symbol displayed on a screen of a display device. apparatus. The symbol adding unit determines a size of a symbol displayed on a screen of a display device based on a time length of a symbol time zone as a process of reflecting the time length. 5. The voice processing device according to 5. The symbol adding unit determines a display color of a symbol displayed on a screen of a display device based on a time length of a symbol time zone as a process of reflecting the time length. 5. The voice processing device according to 5. The voice processing apparatus according to claim 5, wherein the symbol adding unit displays a number indicating a time length of a symbol time zone on a screen of a display device as the process of reflecting the time length. The sound processing device, when sampling the waveform data, performs sampling for each sound source into a microphone for an explainer who reads out the document data, and a microphone for a speaker other than the explainer,
When the voice document corresponding unit extracts a symbol,
Extracting in-document words and symbols from the audio frame sequence sampled through the microphone for the presenter,
The speech processing apparatus according to any one of claims 1 to 8, wherein the speech frame sequence sampled through a microphone for a speaker other than the presenter is extracted as a symbol. The audio processing device, when sampling the waveform data, performs sampling through a microphone array composed of a plurality of microphone elements for specifying a sound source direction,
When the voice document corresponding unit extracts a symbol,
Extracting words and symbols in the document from the audio frame sequence in the sound source direction of the explainer who reads the document data,
The speech processing apparatus according to any one of claims 1 to 8, wherein the speech frame sequence other than the sound source direction of the explainer is extracted as a symbol. The voice processing device further includes a sound source identification unit,
The sound source identification unit calculates the likelihood of each word in the document included in the document data for each sound source direction of the audio frame sequence, and determines the sound source direction having the highest maximum likelihood value as the sound source of the explainer The voice processing apparatus according to claim 10, wherein the voice processing apparatus is identified as a direction. The voice processing device according to any one of claims 9 to 11, wherein the voice processing device displays a symbol on a screen of a display device by changing a display format for each sound source of the symbol. . An audio processing method by an audio processing apparatus for adding sampled waveform data to document data as symbols,
The audio processing device includes an audio frame sequence detection unit, an audio document correspondence unit, and a symbol addition unit,
The voice frame sequence detector
Read the waveform data from the storage unit, create a voice frame sequence by detecting the voice section of the waveform data,
The voice document support unit
By calculating the likelihood for each word in the document included in the document data from the generated voice frame sequence, the first time zone of the voice frame sequence in which the word in the document is spoken, and a word other than the word in the document Detecting a second time zone of the audio frame sequence in which the content is spoken;
The speech frame sequence in the second time zone is extracted as a symbol, and the word in the document uttered in the first time zone that is closer in time to the second time zone in which the extracted symbol is uttered. The position near the position is determined as an additional position of the extracted symbol,
The symbol adding unit includes:
Based on the position of the word in the document read from the storage unit and the addition position of the symbol determined by the voice document corresponding unit, the symbol is arranged in the document data, so that the extracted symbol is the document. An audio processing method characterized by being added to data.   An audio processing program for causing the audio processing apparatus, which is a computer, to execute the audio processing method according to claim 13.

Images