JP2014222290A - Minute recording device, minute recording method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To more highly accurately associate conference voices or conference videos with memoranda recorded as a minute.SOLUTION: A text and a voice are associated with each other based on dissimilarity of the text converted into phonemes on the basis of input time of a memorandum during a conference and utterance time of a voice recorded synchronizing with a video during the conference, and phonemes obtained by voice recognition of the recorded voice. When the text recorded as the conference memorandum is selected, the recorded video or voice is reproduced from the utterance time of the voice associated with the text.

Description

  The present invention relates to a minutes recording apparatus, a minutes recording method, and a program, and is characterized in that, in particular, it is possible to correct a time lag between a voice included in a moving image and text information input from a text input means.

In general, the content of discussions at meetings is an important asset in organizations, and it is often required to submit minutes after the meeting. The meeting had a recording person and recorded the contents of the meeting by hand, which put a burden on the minutes creator. The following technologies have been proposed to assist in the creation of minutes.
In the device described in Patent Document 1, a memo that represents the progress of the conference that is input by the memo editing unit is associated with the conference video based on the date and time when the memo was recorded and the date and time information held in association with the conference video. Based on the conference progress and the audio during the conference, the association of the conference progress with the conference video is retained. The minutes creation unit 3 creates minutes in this associated state, and the created minutes can be viewed by the minutes display unit 5. In this way, a link can be established between the description content of the minutes and the conference video, and it can be confirmed later which scene of the conference the description content corresponds to.

JP 2006-268800 A

  However, as in the apparatus described in Patent Document 1, it is difficult to associate text recorded as a memo with a video with high accuracy only by associating the date and time information with the conference video by the memo editing unit. This is because there is a discrepancy between the date and time when the text was recorded by the memo editing unit and the date and time of the conference video in which the content of the comment corresponding to the text was recorded. Even if the user tries to view the conference video associated with the memo by browsing the link by referring to the memo, the part that the user wants to browse due to the difference between the date and time when the text was created and the video time corresponding to the text Even if you click the note, the video of the time that is not corresponding to the note will be displayed.

  The present invention has been made in view of the above-described conventional example, and provides a minutes recording apparatus and a minutes recording method capable of associating text and video with high accuracy by associating them with each other based on their contents. To do.

The present invention has the following configuration. That is,
Holding means for holding the recorded audio data for each break;
An editing means for storing the input text information for each separator;
An apparatus for recording minutes, comprising: an association unit that associates the division of the audio data with each division of the text information based on the degree of difference of each phoneme, and stores the association.

  According to the present invention, it is possible to associate texts and videos with high accuracy by associating them based on their contents. Further, the text and the video can be displayed in synchronization with each other with high accuracy.

It is a figure which shows the device structural example of this system. It is a figure which shows the block diagram of the text input screen of this system It is the figure which showed the data management structure in this system, and the structural example It is a figure which shows the block diagram of the minutes display screen in this system. It is a figure which shows the whole structure of this system. It is the figure which showed the structure of the synchronous control means part in this system. It is a block diagram which shows the structural example of a speech recognition part. It is a figure which shows the example of a language model. It is a schematic diagram which shows the collation process in a voice collation part. It is a schematic diagram which shows the production | generation process of the text conversion phoneme sequence in a text conversion phoneme sequence production | generation part. It is a figure which shows the processing flow of the synchronous control means in this system. It is a figure which shows the example of a processing result of a synchronous control means. It is a figure which shows the processing flow of the collation part of the synchronous control means in this system. It is a figure which shows the structural example of the display part in this system. It is a figure which shows the example of an image | video reproduction | regeneration by the display part in this system. It is a flowchart of the procedure which browses the minutes.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to FIGS.

[First Embodiment]
In the first embodiment of the present invention, the current character information is added to the input character data and recorded, the sound is recorded at the same time, and the time information of the recorded sound and character data is used in time series. It is a minutes creation / playback device that displays correspondingly. The time information is information for specifying the date and time, such as date and time information.

  Notes taken by minutes and attendees during the meeting are the best material to follow the topic during the meeting. In this embodiment, the entry time (TK) at the time of input is recorded in each sentence of the memo. Completion of input of one sentence can be determined by inputting a line feed code, for example. Before the entry time when the memo was recorded, the topic corresponding to the memo should have been discussed at the meeting. In the minutes creation / playback apparatus of the present embodiment, the audio and video of the meeting is also recorded with time information, and by specifying the part of the note of interest, the audio of the target location linked at the recorded time or You can easily find the picture. Note that information in which video data and a memo are associated with each recording time is referred to as a digital minutes in this embodiment.

<Configuration of minutes creation and playback device>
FIG. 1 is a conceptual configuration diagram showing a configuration of a minutes creation / playback apparatus according to the present embodiment. The minutes recording / reproducing apparatus is configured by connecting a client terminal and an input device such as a video. In FIG. 1, a client terminal 11 is a personal computer, a tablet terminal, or the like, and is a terminal for creating and playing back minutes. In this embodiment, the client terminal 11 includes a CPU, RAM, ROM, HDD, and the like. Note that the description of the present embodiment and the procedure shown in the flowchart are stored in any of the RAM, ROM, and HDD storage units of the client terminal 11, loaded into the RAM, and executed by the CPU. The microphone 12 is a microphone that is connected to the client terminal 11, converts sound into an audio signal, and inputs the sound signal to the client terminal 11. The still camera 13 is a still image shooting camera connected to the client terminal 11. The video camera 14 is a video camera for shooting moving images connected to the client terminal 11. Each of these cameras inputs the captured still image and moving image digital data to the client terminal 11. The digital data of still images and moving images will be called image data and video data, respectively. The video data includes audio data obtained by digitizing an audio signal in synchronization with the video. Also, audio data can be integrated with still image data. When this still image data is reproduced, a still image is displayed and sound is reproduced. The video camera 14 includes a digital camera having a video shooting function, a portable terminal, and the like. Time information is recorded in the video data. For example, it is necessary to be able to specify the date and time in an arbitrary frame. For example, if the shooting start date and time is recorded in the video data and the frame number is recorded for each frame, the date and time can be specified by an arbitrary frame of the video. Conversely, a video frame can be specified from the date and time. Note that audio and text can be associated with each other as with video. In this case, the audio data needs to include information that can specify the date and time at an arbitrary time.

    In the configuration shown in FIG. 1A, character data (text) data is input from the keyboard of the client terminal 11. A text entry time (TK), which is time information at the time of input, is added to the input character data and recorded as text. In this case, the time of input may be a unit of input of character data, or a timing when a line feed is detected. The microphone 12 inputs voice to the client terminal 11 as voice data. Time information at the time of input is added to the input audio data and recorded in a storage unit such as an HDD. Audio data can be added later with time information added. When displaying the minutes, the recorded text is displayed in time series, and the recorded voice is reproduced and output. If a part of the displayed text is designated, the sound corresponding to the part is reproduced.

  In the configuration shown in FIG. 1B, moving image data, that is, video data is input from the video camera 14 to the client terminal 14. Time information at the time of recording is added to the input video data and recorded in a storage unit such as an HDD. As described above, the time information may be added in units of frames, for example. During reproduction, the recorded moving image data is reproduced and output. In the video data, audio data synchronized with the video is recorded.

  In the configuration shown in FIG. 1C, a plurality of client terminals 11 are connected to the server 1 via the network 10. Data input from separate client terminals is processed by the server 1 to reproduce and output moving image data on each client. Thereby, the meeting video and the minutes can be shared by a plurality of users.

  In the present embodiment, an example of processing in the CPU and recording unit of the client terminal 11 is shown. However, in a configuration in which a server exists, a part of the processing may be transferred to the CPU and recording unit of the server. The recording unit is a functional unit that records text in association with video data or audio data, and can be realized by a dedicated processor separate from the main CPU, for example, or by executing a recording program by the CPU. it can.

<Meeting process>
FIG. 5 is a functional block diagram of the minutes recording / reproducing apparatus of the present embodiment. The user inputs a memo from a text input unit 51 such as a keyboard of the client terminal 11. The input memo is added with the entry time (TK) by the timer 53 and recorded by the text recording unit 52. Time information is added to the audio, still image, or moving image input from the audio / video input unit 54 such as the microphone 12, the camera 13, or the video camera 14, and is recorded by the audio / video recording unit 55. At the time of reproduction, the synchronization display control unit 56 synchronizes and the corresponding part is displayed on the display unit 57. A synchronization control method by the synchronization display control unit 56 will be described later.

  Next, the operation when creating the minutes will be described in detail with reference to FIG. 2 and FIG. FIG. 2 is a diagram showing a display screen when the minutes are created by the minutes creation / playback apparatus. This display screen is displayed on the display of the client terminal 11. FIG. 3 is a diagram showing an example of the minutes data recorded in the minutes creation / playback apparatus.

  First, the microphone 12 is connected to the client terminal 11. If the microphone 12 is built in the client terminal 11, it may naturally be used. Note that the microphone 12 connected to the client terminal 11 may not be used if recording is performed using an IC recorder or the like unrelated to the client terminal 11 and audio data is added later. When the operator starts the minutes recording program, as shown in FIG. 2A, an indicator unit 21, a video operation unit 22, a format setting unit 22, and a text input unit 24 are displayed on the screen. Recording starts when the recorder presses the recording start button on the screen. An internal timer starts simultaneously with the start of recording, and the recording start time and the current time are measured. For the purpose of this embodiment, the timer 53 may be a real-time clock that measures the actual time. In this case, it is not necessary to start the timer.

In this state, characters are input to the text input unit 24. When the key is hit with the operation cursor positioned on the text input unit, the input start position is displayed on the left side of the text input unit 24. When a line feed is input in the middle of input, the line moves to a new line and the entry time (TK) at that time is recorded. If further input is made in this state, the time of the input is recorded again. FIG. 2B shows a screen that has been input to some extent. In this example, five sentences have already been entered. When the stop button of the video operation unit 22 is pressed, recording ends and the timer also stops. In this state, if “Save” is selected from the menu, the recorded video data file and text file are saved after inputting the name of the minutes. For example, the file to be saved is saved in a structure as shown in FIG. FIG. 3A is an example of minutes data recorded on December 24, 2008 and saved under the name “Conference 1”. Data is recorded in a folder for each minutes provided in a folder named Data. In this example, the minutes folder is given a name in which recorded dates are arranged. When a plurality of minutes are recorded on the same day, for example, a serial number or time may be added after the date. The following files are included in each minutes folder.
project. dat is a file that records the file structure of the minutes. The file format may be a unique binary format or other text format (XML, JSON, unique, etc.). The file “200812242123456.mp3” in the mp3 format is recorded voice, and the file name is, for example, year, month, day, hour, minute, and second at the start of recording, but it is not necessary to be particular about this. Also, the format may be other formats such as wav instead of mp3. The recording start time may be recorded as a file attribute, for example. The text file “memo.txt” is the text entered in the input field, and is saved as the minutes memo data 302 in the format shown in FIG. That is, a text ID 330 is assigned to each text, and the entry time 331 and the entered text 332 or text 333 are recorded in association with each text ID. The text 332 is a summary memo, and the text 333 is a detailed memo (simply called a memo) belonging to the summary memo. The distinction between the summary memo and the detailed memo is based on, for example, the input operator. Here, the memo described by the person in charge of making minutes during the meeting is used as the summary memo, and the memo described by other attendees is used as the memo.
By saving the memo in the plain text format in this way, it can be edited later by another editor or the like. If the generality of data is not taken into consideration, another format such as binary may be used.
The above file structure is merely an example, and all the files in the minutes folder may be one file. For example, one memo may be divided into a plurality of files and input times may be associated with each text file, for example, text input may be completed at a period without a line feed code. This makes it possible to reduce the time difference between the start and completion of input for long memos. The text may be in other forms.

  By connecting the video camera 13 as shown in FIG. 1B, not only audio but also video can be recorded simultaneously. This screen example is the same as that shown in FIG. However, once a character is written on a new line and the time is recorded, even if the character is deleted, the recorded time does not disappear. If nothing is written on a new line and another line break is made, the time is not recorded in association with a memo with only a line break. This feature can also be applied to audio-only minutes.

  When a video file taken with a video camera or the like during the creation of the minutes is included in the minutes file, the file structure of the minutes data 301 is as shown in FIG. The minutes including the video file have a structure in which a metadata file of video data “movie.meta” and a folder “movie” in which the video file is recorded are added to the structure of FIG. Have Here, the shooting time can be extracted from the information recorded in the video file. Information on the shooting time (shooting time) is used by the synchronization control unit 56 that synchronizes the video file and the text file. Note that the moving image file of this embodiment includes not only video but also audio data. The video data and the audio data are synchronized, and not only the video but also the time when the audio is recorded can be specified from the video file.

<Synchronous control>
Here, a detailed description of the synchronization control unit 57 will be given with reference to FIGS. FIG. 6 is a block diagram showing an embodiment of the synchronization control means 57 for associating input speech and text. The synchronization control means 57 of this embodiment includes a speech recognition unit 61, a recognition result phoneme string storage unit 62, a text conversion phoneme string generation unit 63, a text conversion phoneme string storage unit 64, and a combining unit 65.

  The purpose of this synchronization control unit is to input the time when the text information is spoken with respect to the entry time information (TK) associated with the text information stored in the format of FIG. Voice recognition is performed on a voice (including voice synchronized with data), a time when voice is recorded (referred to as speech time) is detected, and new speech time information (TH) is stored together with text information. The utterance time information (TH) may be additionally recorded in the format of FIG. 3C, or the entry time (TK) may be overwritten. However, when additionally recording, it is necessary to clearly indicate the association between the added utterance time information (TH) and the text. For this purpose, for example, in the case of a table format as shown in FIG. 3 (c), it may be recorded on the same line as the text of the corresponding memo.

  The configuration of the synchronization control unit 56 is shown in FIG. By reproducing the audio and video recorded in the audio video recording unit 55, input of audio data to the audio recognition unit 61 is started. When the voice data is directly input, the voice data as it is is input. When the input is video data, the voice data uttered by the participant is input together with the conference video. Therefore, the video and audio are acquired without time lag. The voice data acquired in this way is input to the voice recognition unit 61. The voice recognition unit 61 recognizes input voice data and outputs a recognition result phoneme string and utterance time (TH) corresponding to the voice data. The output data is input to the combining unit 65. Here, the utterance time (TH) is obtained by calculating the time when the recognition result phoneme string was uttered based on the time information added to the sound (video).

<Voice recognition unit>
FIG. 7 is a block diagram illustrating a configuration example of the voice recognition unit 61. The speech recognition unit 61 includes a speech detection unit 71, an acoustic analysis unit 72, an acoustic model storage unit 73, a language model storage unit 74, and a speech collation unit 75. This configuration is common in speech recognition, but will be briefly described.

  The voice detection unit 71 cuts out the voice of the section including the human voice from the input voice data and sends it to the acoustic analysis unit 72. For example, a voice detection method based on the magnitude of input power can be used to cut out the voice in the voice detection unit 71. In this voice detection method, the power of the input is calculated sequentially, and the time when the power of the input exceeds a predetermined threshold continuously for a certain period of time is determined as the start time of one section of the voice. The time point when the predetermined threshold value is continuously decreased for a certain period of time is determined as the end time point of the speech segment. The voice cut out by the voice detection unit 71 is sequentially sent to the acoustic analysis unit 72 from the voice start time to the voice end time.

  The acoustic analysis unit 72 performs an acoustic analysis of the voice cut out by the voice detection unit 71, and sends an acoustic feature quantity sequence representing a voice feature such as a mel frequency cepstrum coefficient (MFCC) to the voice matching unit 75.

  The acoustic model storage unit 73 stores a standard pattern such as a hidden Markov model (HMM) prepared for each phoneme that is a unit constituting Japanese speech. A standard pattern corresponding to any Japanese word / sentence can be created by connecting this standard pattern in accordance with the phoneme sequence constituting the Japanese word / sentence.

  The language model storage unit 74 stores a language model that defines connection relationships such as between Japanese words and phonemes. This language model includes (1) a continuous syllable recognition grammar that prescribes the connection relationship between syllables, (2) a grammar rule that prescribes a connection relationship between words, and (3) a probability that a set of N phonemes continues. There are a statistical language model for defining, and (4) a statistical language model for defining the probability that a set of N words continues.

  FIG. 8 is a diagram illustrating an example of a language model. FIG. 8 (a) is a continuous syllable recognition grammar that defines the connection relationship between syllables, and this shows the connection relationship between consonants / b // d /... And vowels / a // i /. It prescribes. FIG. 8B is a grammatical rule that defines a connection relationship between words, which defines a connection relationship of / word 1 // word 2 /.

  FIG. 9 is a schematic diagram showing a collation process in the voice collation unit 75. In FIG. 9, the acoustic feature quantity sequence sent from the acoustic analysis unit 72 is collated with the standard pattern by the voice collation unit 75, and as a result of collation, / sh // i /... / U / is obtained. This indicates that the start and end times of the speech section corresponding to each phoneme are acquired. The utterance time (TH) at which the utterance is started is determined based on the time information of each phoneme.

  Returning to FIG. 7, the voice collation unit 75 generates a standard pattern by connecting the acoustic model in accordance with the connection rules described in the language model, and uses the Viterbi algorithm to transmit the acoustic feature amount from the acoustic analysis unit 72. Match columns against standard patterns. As a result of this collation, the correspondence between the voice section that maximizes the collation score of both and the standard pattern is obtained. As a recognition result in the speech recognition unit 71, a start time and an end time of the speech section corresponding to each phoneme constituting the recognition result phoneme string and the standard pattern are acquired. The recognition result phoneme string obtained as a result is stored in the recognition result phoneme string storage unit 62. Voice collation is described in “Seiichi Nakagawa et al .:“ Voice Recognition by Stochastic Model ”” The Institute of Electronics, Information and Communication Engineers. The phoneme string can be represented by, for example, a chord string indicating each phoneme. The time corresponding to the phoneme is recorded in association with the phoneme code.

  The recognition result acquisition process of the voice recognition unit 61 is performed when instructed by the combining unit 65. At that time, the phoneme which is the collation intermediate result (partial collation result) obtained in the collation process and its start time (or end time) are also stored in the recognition result phoneme string storage unit 62 until an acquisition instruction is given. Shall be retained. In this example, this start time corresponds to the utterance time of the phoneme. Then, according to the instruction from the combining unit 65, the verification is continued by taking over the verification intermediate result obtained when the previous recognition result was obtained. Alternatively, the collation is restarted from the initial state after discarding the collation intermediate result used at the previous collation.

  On the other hand, the text converted phoneme string generation unit 63 generates a text converted phoneme string corresponding to the text stored in the text recording unit 52. The unit of text input here is a unit input from the text input unit 51, which may be a single character unit or a text unit. In this embodiment, the next line feed code from the first character or the next line feed code to the next line feed code. The description will be made with the unit as the unit. The text converted phoneme string is stored in the text converted phoneme string storage unit 64.

  FIG. 10 is a schematic diagram showing a text conversion phoneme string generation process in the text conversion phoneme string generation unit 63. The text conversion phoneme string generation unit 63 performs morpheme analysis of the text described by the kana-kana mixed sentence, divides it into parts of speech and converts it into a kana character string representing a reading, and further converts kana characters into phonetic symbols. The conversion table describing the conversion rules is converted to a phoneme string to generate a text-converted phoneme string. The generated phoneme string has the same format as the phoneme string output by the speech recognition unit 61.

  For example, when the text string described by the kanji kana mixed sentence is “7 o'clock news”, the text conversion phoneme string generation unit 63 first performs “7” “time” “no” “news” by morphological analysis. ”And“ is ”. Next, these are converted to the kana character strings "Shi", "ji", "no", "news", and "is" representing the readings, and a conversion table that describes the conversion rules from kana character strings to phonetic symbols Referring to the kana character string as a phoneme string / sh // i // ch // i // j // i // n // o // ny // uu // s // u // d // Convert to e // s // u /. This phonetic symbol corresponds to a chord indicating a phoneme.

  The combining unit 65 collates the recognition result phoneme sequence stored in the recognition result phoneme sequence storage unit 62 with the text conversion phoneme sequence stored in the text conversion phoneme sequence storage unit 64, and associates speech with text. Combine as a result. This process will be described in detail with reference to FIG. Schematically, the recognition result phoneme string and the text-converted phoneme string are combined based on the approximation between the text entry time and the speech utterance time and the phoneme string match.

  The phoneme string collation result storage unit 66 holds the collation intermediate result (partial editing distance) obtained by the combining unit 65 during the collation process. If a plurality of texts correspond to one voice as a result of collation, all the plurality of results are stored. By storing these results, even when the same word is spoken or when the same text is input, it is possible to associate without problems.

<Join processing>
Here, processing when the combining unit 65 is realized by a program will be described in detail with reference to the flowchart of FIG. When the combining unit 65 starts processing, first, the CPU (hereinafter referred to as CPU) of the client terminal 11 extracts an all-text converted phoneme sequence list stored in the text converted phoneme sequence storage unit 62 (S1101). Next, the CPU starts reproducing the conference audio (video) stored in the audio video recording unit 55 in order to generate a phoneme string (S1102). When the voice is reproduced, the CPU extracts a recognition result phoneme string generated from the voice recognition unit 61 (S1103), and uses it as a target recognition result phoneme string and compares it with a text converted phoneme string (S1104). The recognition result phoneme string includes the phoneme string and the utterance time of each phoneme as described in FIG. The comparison target in the text converted phoneme string at this time is a text converted phoneme string having an entry time (TK) after the utterance time (TH) of the recognition result phoneme. For example, in the example of FIG. 12A, the text-converted phonemes 3 and 4 having the entry time TK after the utterance time TH = 0: 40 of the target recognition result phoneme string “shinkai” are to be compared. The result of the comparison is output as the degree of difference defined according to the number of operations of insertion, deletion, and replacement of phonemes necessary for converting the first phoneme string to the second phoneme string. In step S1104, the degree of difference between the target recognition result phoneme string and the text-converted phoneme string to be compared is obtained. In FIG. 12A, the dissimilarity is 1 for the text converted phoneme 3 and the dissimilarity is 0 for the text converted phoneme 4. Next, the CPU determines whether there is a text-converted phoneme string having a degree of difference equal to or less than a predetermined value from the text-converted phoneme strings that are to be compared with the recognition result phoneme string of interest (S1105).

  If there is a corresponding text converted phoneme string, the speech time (TH) of the target recognition result phoneme is associated with the corresponding text converted phoneme string (S1106). This is intended to detect the speech utterance time (or video time) of the content close to the content of the text, and is an example of detecting a portion where the content is close while comparing the text and the speech. Note that, here, since the text-converted phoneme string corresponding to the phoneme string that is the voice recognition result is detected as a reference, the target voice recognition phoneme string does not necessarily match the memo unit. Therefore, the utterance time associated with the text in S1106 is the utterance time of the phoneme at the beginning of the text (in this example, the line feed code is a delimiter), and is recorded in association with the utterance time of the text.

  When the comparison is completed, it is determined whether or not the sound (moving image) has been reproduced to the end (S1107). If not, the process branches to S1103 and proceeds to the determination of the next speech recognition result. Repeat until finished. If the next recognition result phoneme can be extracted earlier than the processing of S1104 to S1106, the reproduction of the sound (moving image) may be temporarily stopped, and the reproduction may be started after S1106 ends.

  After these associations are completed, the combining unit 65 also determines whether a plurality of text converted phonemes are associated with one recognition result phoneme, and a plurality of recognition result phonemes is associated with one text conversion phoneme. When the columns are associated, it is determined which recognition result phoneme sequence has the first priority. In this determination, for example, the phoneme string having the latest utterance time is given first priority. However, among the text recognition phoneme strings having a degree of difference equal to or less than the threshold, the one with the smallest degree of difference may be set as the first priority, and the first priority based on the utterance time may be determined when there are a plurality of text recognition phoneme strings.

<Calculation of dissimilarity>
In order to calculate the degree of difference in step S <b> 1104, the combining unit 65 compares the recognition result phoneme sequence stored in the recognition result phoneme sequence storage unit 62 with the text converted phoneme sequence stored in the text converted phoneme sequence storage unit 64. . As a result of the comparison, a difference degree indicating the degree of difference between the two is calculated. For example, the first phoneme string “/ sh // i // N // k // a // i /” is replaced with the second phoneme string “/ t // o // k // a // i /”. In the case of deformation, the procedure is required at least three times as shown below, so the difference is 3.

1. / Sh / / i / / N / / k / / a / / i /
2. / T / / i / / N / / k / / a / / i / (Replace "/ sh /" with "/ t /")
3. / T / / o / / N / / k / / a / / i / (“/ i /” replaced with “/ o /”)
4). / T // o // k // a // i / (deletes “/ N /” and ends).

  The simplest procedure for obtaining the degree of difference is associating phonemes in order from the top, and counting the number of different phonemes until any phoneme string ends. If any phoneme string is long, the number of phonemes that is the difference is added to the number of different phonemes counted. The obtained result is the degree of difference. Of course, this is an example, and other methods could be applied.

<Text-to-speech list>
When the processing of FIG. 11 is completed, a list 1202 including a recognition result phoneme sequence associated with each text-converted phoneme sequence as shown in FIG. It is created in the column combination result storage unit 66. The input text is assigned a text ID, the text converted phoneme string, the entry time (TK) of the text, the associated recognition result phoneme string and its ID, and the utterance time of the recognition result phoneme string, Are stored in association with each other. Here, the flag OnClick indicating that the text-converted phoneme is associated with the first priority and the flag IsSame indicating that the recognition result phoneme string is associated with the previous text are described together with the procedure of FIG. To do. The text-speech correspondence list 1202 can also be referred to as an association storage unit that stores associations between text breaks and speech breaks.

<Decision of first priority>
The procedure for determining the first priority will be described with reference to FIG. This procedure is executed after the procedure of FIG. First, a comparison result is extracted from the text-speech correspondence list in FIG. 12B (S1501). Next, a determination is made as to whether or not there is one recognition result phoneme for the extracted text converted phoneme and whether the recognition result phoneme has already been reached (S1502). In the example of FIG. 5 is a recognition result phoneme sequence associated with N.5. Since only 1, the flag OnClick indicating the first priority is set to TRUE. In addition, the recognition result phoneme string N. Since 1 is not above the focus column of the list 1202, the flag IsSame indicating that the ID of the same recognition result phoneme string exists above the focus column of the list 1202 is set to FALSE. In other words, IsSame is a flag indicating that it is a recognition result phoneme string that is extracted from the list 1202 and redundantly associated with the tested text.

  Thereafter, the process proceeds to S1504, where it is determined whether or not a set of all text converted phoneme strings and recognition result phoneme strings have been collated. This set is described as text-speech, and when referring to FIG. 12, it is indicated by a set of Tx-Ny and each identifier. If there is an unprocessed text-speech, S1501 and S1502 are performed again. Next, text T., which is text-speech retrieved from list 1202. The recognition result phoneme string associated with N. 2 and N.I. Therefore, the most recent time information among the recognition results is given the first priority (S1503). In this case, the recognition result phoneme string N.I. 3 is the recognition result phoneme sequence N.3. 2 is newer than the utterance time of 2. 3 flag OnClick is set to TRUE, and the recognition result phoneme string N. 4 OnClick is set to FALSE, and T. 6 does not exist, the recognition result phoneme string N. 3, N.I. Each of the four IsSame is FALSE. This process is referred to as text T.P. 7 in the same manner, the recognition result phoneme string N. 2, N.I. 3 flag OnClick is FALSE, and the recognition result phoneme string N. The flag OnClick of 4 is set to TRUE. The text N.I. 4 is the first appearance, so the flag IsSame is false (FALSE) and the text N.4 is displayed. 2, N.I. Since 3 has already appeared in association with the above-mentioned text T6, it is set to true (TRUE).

  If the processing has been completed for all the text-speech in the list, the process branches to S1504 to determine whether or not a plurality of text information phonemes is the first priority of one recognition result phoneme. That is, it is determined whether or not there is one recognition result phoneme that is associated with a plurality of text-converted phoneme strings in the list and that has a flag OnClick of TRUE for the plurality of text-converted phoneme strings.

  For this purpose, first, the comparison result so far is referred to (S1505), and the value of OnClick for each recognition result phoneme string ID is read (S1506). At this time, the text-to-speech OnClick flag is referred to in the order of the recognition result phoneme string ID having the largest number, that is, the one having the later utterance time, among the text conversion phoneme string ID having the larger number. In the example of FIG. 12B, the recognition result phoneme string IDs N2 and N3 associated with the text T7 are also associated with the text T6. However, since none of the flags OnClick is TRUE for a plurality of text converted phoneme strings, the process proceeds to S1509 and the same processing is executed until all text IDs are collated.

  On the other hand, in the processing of S1506, when the contents of the list are as in the example of FIG. 12C, the recognized recognition result phoneme string N2 is flagged for the text converted phoneme strings T13, T12, and T11. OnClick is set to TRUE. This means that the text converted phoneme string T.P. 11, T.W. 12, T13, the recognition result phoneme string N.I. 2 indicates that the first priority is selected. Therefore, the process proceeds to S1507. In S1507, it is determined whether there is an identical recognition result phoneme string in which the flag OnClick is TRUE after the reference recognition result phoneme string. If there is, the value of the flag OnClick corresponding to the reference recognition result phoneme string is rewritten to FALSE (S1508). At the same time, among the recognition result phoneme strings associated with the text, the OnClick value corresponding to the recognition result phoneme string ID immediately before the recognition result phoneme string in which the OnClick value is rewritten to FALSE is changed to TRUE. The rewriting of the two OnClick flags is collectively referred to as updating the OnClick flag.

  Note that in S1507, the same recognition result phoneme string having the flag OnClick as TRUE is present in the list 1202 after the reference recognition result phoneme string, and a plurality of recognition result phoneme strings are included in the target text-speech. It may be determined that is associated with one text. This is because when only one recognition result phoneme string is associated, if this is removed from the first priority, there is no other candidate to be the first priority. Therefore, in this case, if a plurality of recognition result phoneme strings are associated with each other, the process branches to S1508 to update the text-speech OnClick and the previous text-speech OnClick.

  On the other hand, if the condition of S1507 is not satisfied, the process branches to S1509. In the example of FIG. 11, the recognition result phoneme string N2 associated with T12 has a flag OnClick of TRUE in the text conversion phoneme string T13 that appears thereafter, and therefore branches from S1507 to S1508 and the value of OnClick is updated. Since there is not one recognition result phoneme string associated with the text, even if this condition is given as a logical product, the process branches to S1508. That is, the corresponding OnClick flag is rewritten to FALSE. At the same time, the OnClick flag of the recognition result phoneme string N1 (that is, T11-N1 and T12-N1) associated with each text is rewritten to TRUE.

  On the other hand, the text converted phoneme string T.E. In the recognition result phoneme string N2 associated with No. 13, since the recognition result phoneme string N2 whose OnClick is TRUE does not appear thereafter, S1508 is skipped, and the value of the flag OnClick is maintained as it is.

  FIG. 12D shows a state where the first priority text is determined by the procedure of FIG. 13 for the recognition result phoneme string N2. In this state, the T.S. The next T.13. The 12 recognition result IDs are compared with the OnClick value of the same recognition result ID in T11. Even in this state, T.P. 12-N. 1 and T.W. 11-N. 1, since a plurality of OnClicks are TRUE, the process proceeds to S1507. Since there is no number smaller than 1, the process proceeds to S1509. The process proceeds to 11 recognition result ID processing, and the processing ends.

  As shown in FIG. 12C, even after these processes are executed, the same recognition result phoneme may be selected with a first priority from a plurality of text-converted phonemes. There is no effect on the operation of this system. As described above, the list in which the text and the recognition result phoneme string are associated is completed. This list is also a list for associating the audio minutes or video minutes, which are the basis of the recognition result phoneme sequence, with the text entered as a memo. Since the text-to-speech correspondence list 1202 is created for each minutes, when a plurality of minutes are stored, a folder for each minutes and a unique identification name are added to distinguish the minutes. The

<Reproduction of digital minutes>
Next, with reference to FIG. 4, FIG. 14, and FIG. When the operator gives an instruction to start the reproduction program, the CPU starts the browsing program. Actually, the playback mode and the recording mode may be set as one program, and the mode may be selected at startup. Alternatively, it may be activated first in the playback mode and changed to the recording mode when necessary. When started, the minutes reproduction program refers to the text-speech correspondence list 1202 and the minutes data 301 and minutes memo data 302 of FIG. 3, as shown in FIG. A user interface screen including a section (if there is a document file) 42, a text display section 43, a video scene display section 44 (if there is a video file), and a summary display section 45 is displayed. However, the example contents of the text-to-speech correspondence list 1202 in FIG. 12, the example contents of the minutes memo data 302 in FIG. 3, and the example contents in FIGS. 4 and 14 are different examples and are not related.

  When displaying the user interface, the time is first displayed on the time display unit 41 in time series, and the materials associated with the time are displayed on the material display unit 42 in time series for each displayed time. For this reason, for example, by assigning time information to each page of the conference material, it can be associated with a video frame photographed at the same time or a recorded memo. Then, referring to the minutes memo data 302, the summary memo 332 and the memo 333 are displayed in association with the displayed timeline based on the entry time. The summary memo 332 is also displayed on the summary display unit 45 in the order of entry time. Further, the video scene display unit 44 displays a frame of the shooting time in association with the time displayed on the time display unit 41. For example, if a time scale is displayed on the time display unit 41, the frame to be displayed may be selected according to the time corresponding to the scale.

  When the user interface 401 is displayed in this manner, the user can reproduce the video corresponding to the memo by operating the user interface 401.

  First, when one line of the summary display unit 45 is clicked, the CPU finds a recognition result phoneme string associated with the text ID of the clicked text with reference to the text-to-speech correspondence list 1202 and includes OnClick. The utterance time (TH) whose flag is TRUE is read, and the stage including the utterance time is highlighted in the user interface 401. For example, the text in the text display unit 43 corresponding to the clicked summary text and the time of the corresponding time display unit 41 are highlighted with different colors or blinking. Of course, the corresponding material or frame may be highlighted.

  When one material in the material display unit 42 is clicked, the clicked material is enlarged and displayed on the screen of the client terminal 1. When the text in the test display section 43 is clicked, the time of the utterance time (TH) at which OnClick is TRUE among the recognition result phonemes associated with the text as shown in FIG. Audio or video is reproduced by the CPU. This association is performed as follows. For example, among the recognition result phoneme strings corresponding to the text ID of the clicked text, a recognition result phoneme string whose OnClick flag is TRUE is found from the text-speech correspondence list 1202. As a result, the utterance time (TH) can be specified, and the video is reproduced from the video data included in the minutes data 301 from the frame recorded at the utterance time. Similarly, even when one video (frame) in the video scene display unit is clicked, the video is reproduced starting from that frame. An example of this screen is shown in FIG. At the bottom of the video screen 1501, a timeline 1502 indicating the playback point in the entire video, a rewind button 1503, a pause button 1504, and a fast forward button 1505 are displayed. During the pause, a play button is displayed instead of the pause button 1504. This display is displayed in a separate window from the user interface 401, for example.

  During video playback, a mark indicating the utterance time of another associated recognition result phoneme may be displayed on the timeline 1502 as a playback candidate. The reproduction candidate refers to the text-to-speech correspondence list 1202, selects a recognition result phoneme string whose OnClick flag is TRUE, associates it with the phoneme string, and displays the utterance time of the phoneme string on the timeline 1502. A mark 1506 (diamond in the example of FIG. 15B) is displayed at a position corresponding to. When this mark is clicked, video playback starts from the utterance time of the phoneme string associated with the clicked mark. If the cursor is placed on a mark, text associated with the phoneme string associated with the mark may be displayed with a balloon or the like with reference to the text-speech correspondence list. In addition, when displaying the reproduction candidate, it is possible not to display the recognition result phoneme string in which IsSame is TRUE. This is because the recognition result phoneme string associated with the same text-converted phoneme in the time when the entry time TK is early is not displayed.

  It also has a search function. When the search window 46 in the user interface 401 is selected, a search word can be input. Clicking the search button highlights the location where the relevant text exists.

  Next, an outline of a procedure for browsing the minutes described above will be described with reference to FIG. This procedure is a flowchart of the above-described operation. First, the user operates the minutes recording / reproducing apparatus and inputs an instruction to reproduce the digital minutes on the displayed user interface 401. When the user browses the minutes, the user moves the operation cursor and presses the operation button in accordance with the location to be browsed. The position of the operation cursor when the operation button is pressed is determined (S1601), and if it is the summary display unit 45, the display column corresponding to one line of the memo in the summary display unit 45 is emphasized (S1602). On the other hand, if the position of the operation cursor is the text display unit 43, the video is reproduced from the utterance time of the recognition result phoneme string (limited to OnClick = TRUE) associated with the text (S1603). Further, if the position of the operation cursor is the video display unit, the video is reproduced from the designated frame (S1604).

  In this way, the memo recorded by the person in charge of the minutes can be viewed while automatically matching the video and audio of the conference, and the topic at that time can be easily followed from the video or the like. By specifying a part of the memo created by the person in charge of the minutes, the part corresponding to the part can be reproduced from the video and audio of the meeting, and it becomes easy to find the part to be heard. Voices recorded by other means can be used, and can be used when a recording device fails. You can search not only the audio of the meeting but also the video, and the contents will be easier to understand even in a meeting that is explained while drawing a picture on the whiteboard. It is possible to display not only the notes created by the person in charge of the minutes but also data such as materials used at the time of the meeting, which makes it easy to convey the contents of the meeting.

As described above, in the embodiment of the present invention, the minutes creation / playback apparatus adds the time information when the input character data is input and records it as text, and the recorded voice or video. Associate with. As a result, not only the input time of the text but also the text and the voice can be related by their phonemes. When a memo is designated, the voice associated with the memo text by phoneme can be reproduced in the designated memo, and the text and the voice or video can be associated with higher accuracy. In addition, since the corresponding position of the character data is displayed at the time of reproduction, it is possible to create a minutes that can easily refer to the target portion.
[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

Holding means for holding the recorded audio data for each break;
An editing means for storing the input text information for each separator;
An apparatus for recording minutes, comprising: an association unit that associates the division of the audio data with each division of the text information based on the degree of difference of each phoneme, and stores the association. The holding means holds the voice data in association with the date and time information spoken for each break,
The editing means stores the text information in association with the date and time information input for each break,
The associating means selects the text information delimiter based on the degree of phoneme difference from the audio data delimiters associated with the date and time information earlier than the date and time information associated with the text information delimiter. The minutes recording apparatus according to claim 1, wherein the minutes recording is associated with voice data delimiters.   In the case where the association means can associate a plurality of audio data delimiters with one text information delimiter, the associated date and time information is the newest audio data delimiter as the one text information delimiter. The minutes recording apparatus according to claim 1, wherein the minutes recording apparatus stores the information in association with each other.   4. The text information delimiter is a character string sandwiched between line feed codes, and the audio data delimiter is a phoneme string recognized from the audio data. Minutes recording device described in 1.   5. The minutes recording apparatus according to claim 1, wherein the audio data is audio data recorded in synchronization with video data.   6. The minutes recording apparatus according to claim 1, further comprising user interface means for displaying the text information in chronological order for each delimiter. The audio data is audio data recorded in synchronization with video data,
7. The minutes recording apparatus according to claim 6, wherein the user interface means further displays a part of frames constituting the video data in time series.   The user interface means further includes a date and time further associated with the audio data delimiter associated with the selected text information delimiter when one of the displayed delimiters of the text information is selected and playback is instructed. 3. The minutes recording apparatus according to claim 2, wherein the audio data is reproduced from information. The audio data is audio data recorded in synchronization with video data,
9. The user interface means further reproduces the video data from a point in time indicated by date and time information further associated with the audio data delimiter associated with the selected text information delimiter. Minutes recording device described in 1.   When reproducing the video data, the user interface means specifies a timeline for specifying the time point of reproduction in the video data, and audio data associated with each of the text information delimiters in the timeline. A mark indicating date and time information associated with each delimiter is further displayed, and when the mark is instructed, the video data is reproduced from the time indicated by the date and time information associated with the instructed mark. The minutes recording device according to claim 9. A holding step for holding the recorded audio data for each break;
A holding process for storing the input text information for each delimiter;
A minutes recording method, comprising: an associating step of associating each of the text information delimiters with the delimitation of the audio data based on a degree of difference of each phoneme and storing the association. A holding step for holding the recorded audio data for each break;
A holding process for storing the input text information for each delimiter;
A program for causing a computer to execute an associating step of associating each of the text information delimiters with the delimitation of the audio data based on the degree of difference of each phoneme and storing the association.

Images