JP2003274345A - Multimedia recording device, multimedia editing device, recording medium therefor, multimedia reproducing device, speech record generating device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a speech record generating device which automatically generates a speech record of a meeting of a plurality of members. <P>SOLUTION: The device has a means (AA) of making a source selected read of data from a medium where audio and video from pieces of input data and time information are digitally recorded, a means of restoring audio data, a means of detecting a speech break position, a means (B1) of dividing a voiced part into phrase units, a speech recognition means (C1) of making a phrase unit voice into text data, and a means (D1) of adding the time information to the text data in the phrase units, has a means of selectively reading another source out of the medium by the means (AA), a means of restoring speech data, a means of detecting a speech break position, a means (B2) of dividing a voiced part into phrase units, a speech recognition means (C2) of making a phrase unit voice into text data, and a means (D2) of adding the time information to the text data in the phrase units, and has a means (F) of alternately arraying the pieces of text data generated by (D1) and (D2) in the time order, and an output means (G). <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia recording device, a multimedia editing device, and a recording medium, a multimedia reproducing device, and a memo recording device for automatically creating a memo in a conference in which a plurality of people participate. .

[0002]

2. Description of the Related Art Heretofore, it has not been possible to find any publicly known material in Japan that transcribes a statement made by a plurality of people. Only 20
At the ATR Spoken Language Communication Research Laboratories held in May 2001, HSC2001, International Workshop of Hands-
Free Speech Communication), "The meeting browser developed by Carnegie Mellon University
A prototype report has been made as "a system that can automatically create simple minutes" (http://www.is.cs.smu.edu/js/meeting.html). In this method, one omnidirectional (360 degree) camera is installed in the center of the conference table, and it is judged by the image processing that the person with the movement of the mouth of the participant is speaking, and the whole voice is input in common. Sort,
Speech recognition processing is performed to create English characters.

What is new in this approach is "360
This is a combination of “selecting a moving person from a plurality of persons based on the camera input” and “voice recognition”.
Around 1988, Intel Corporation had already implemented it, but it did not become popular because there was no application application other than some military applications. Further, as a method of selecting a speaker from a plurality of persons, as a method of determining a mouth from a fuzzy video image, “Image transmission device” in Japanese Patent Laid-Open No. 8-317363, and in that document, Is also known and has no novelty. Also, since its performance is a prototype, it may not be in the evaluation stage, but the microphone is not a "narrative type" dedicated to each speaker, but a shared open type for several people, or Since it is assumed that it is a "microphone array type," the current technology cannot expect much about the voice recognition system.

On the other hand, as an approach for taking a memo with an image of a conference pattern instead of creating a minutes, Japanese Patent Laid-Open No. 2001-2001
The "Dialogue Recording System" of Japanese Patent Publication No. 211440 is known, in which a camera is mounted on the head of a person to record the contents spoken in the corridor, and the contents can be re-viewed later to be made into an idea, or during conversation. Only the inspirational part is selected and recorded. According to this, feeling that a dialogue is recorded may affect human communication. Therefore, it is approached to provide a portable dialogue recording device.

Practical use of the approach of accurately creating minutes letters from conversational speech was finally implemented last year. NH
The K news program is transcribed "for the hearing impaired", and its method is disclosed in Japanese Unexamined Patent Publication No. 2001-166790, "Transcribed text automatic generation device, voice recognition device and recording medium". This method is applied to the announcer, who is a person who pronounces properly, but the recognition accuracy is extremely good. The drawback is "ambiguity" about the words spoken by the general public
There is no correspondence to, but it is not a serious drawback.

As another continuous-speaker speech recognition technique, "language modeling system and method for forming language model" in Japanese Patent Laid-Open No. 6-318096 is excellent. This is an improvement of the method of using a language (syntax) model to improve the probability of judgment when recognizing the pronunciation content of a word by words, and it has become possible to reduce the amount of computer memory required by a conventional language model. is there. However, this method is a so-called “powerful method”, as IBM itself reads, it is an approach that competes in terms of the amount of pattern matching dictionary and the amount of language syntax rather than the algorithm. This is put to practical use and applied to speech recognition software for PC and its syntactic language model.

Regarding multi-channel simultaneous recording of video and audio of a conference pattern, conventional simultaneous recording has been disclosed in Japanese Patent Laid-Open No.
In “Program Information Providing System, Program Information Providing Device, and Recording / Playback Control Device” of 0-217063, a method of setting a bit rate of a content when simultaneously recording a plurality of contents in the same time zone of digital broadcasting is proposed. . This makes effective use of a recording device having insufficient performance and is applied to recording a digital broadcast program having a large amount of data. Further, in the cited document of the present invention, Japanese Patent Application Laid-Open No. 10-2
43303, Japanese Patent Laid-Open No. 7-21619, and a method of recording one VTR tape in common is also examined in Japanese Patent Laid-Open No. 9-307846, and in addition, DISK.
As a technique applicable to a recording medium such as JP-A-7-1
07461, Japanese Patent Application Laid-Open No. 11-98478 proposes a "video encoding technique for re-compressing a video that has been coded and compressed".

As the most practical method of recording video from a plurality of sources, in the "video device" of Japanese Patent Laid-Open No. 2001-8144, the NTSC signal using the HDD is MP
There is a proposal to convert the EG2 signal and record, simultaneously play back, and simultaneously record two sources. The present invention structure itself has been implemented as a "VIVO recording system" by using a video card with a built-in TTV tuner made by ATI in this application (11th year of the year) as a PC-based recording method in the United States. Is scarce. Also, stripe recording (R
AID-level 2) is similarly lacking in novelty. However, the configuration operation is highly realistic, and has appeared in the Japanese market since the spring of 2001 as an HDD recording device.

[0009]

As described above, in recent years, there have been very few active approaches to characterize the content of comments at general meetings. Also, there are few approaches to record the pattern of the meeting with video. However, due to advances in CPU, memory circuit technology, and large-capacity storage medium technology in the world, it is possible to reduce the size of the device, and in recent years, the communication infrastructure is rapidly increasing in speed and becoming cheaper IP, and there is no space for installing equipment. That is not an excuse, let alone the approach that the recorded video affects the conversation, for example, after the first boom in the use of video conferencing in 10 years.

Further, the proposal of the configuration using the VTR and the configuration using the HDD are both studies for the purpose of recording and reproducing the video entertainment, and the quality of the input source should not be lowered as much as possible. This is a study for the purpose of playing back and enjoying the recorded contents by recording, or recording for a long time. That is, there is no correlation between the contents of a plurality of sources, which is a proposal on the premise that there is no device for processing a correlation between two sources.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a comment recording apparatus for automatically generating a comment in a conference in which a plurality of people participate.

Another object of the present invention is to allow a party or a third party of the conference to see and hear the text together with the text characters of the transcript when the pattern of the conference is reproduced.

In order to achieve such an object, the invention according to claim 1 includes means for managing date and time,
It is characterized in that it has a plurality of input channel means for simultaneously inputting a plurality of digital data of video synchronized with audio and has means for digitally storing digital data and date / time information of each channel in a recording medium.

The invention according to claim 2 has a plurality of means for managing the date and time and a plurality of input channel means for simultaneously inputting a plurality of digital data of the video synchronized with the voice, and the digital data of each channel and the date and time information are recorded on the recording medium. It is characterized in that it is recorded by a multimedia recording device having means for digitally storing the data.

According to a third aspect of the present invention, means (A) for reading data from a medium in which audio / video and date / time information are digitally recorded, means for restoring audio data, means for detecting an audio interruption position, and audio A voice recognizing means (C) for converting the voiced part into phrase units and converting the phrase unit voice into text data; and a means (D) for additionally creating the date and time information in phrase unit text data. Means for restoring video data (V) and means for outputting (G)
It is characterized by having and.

According to a fourth aspect of the present invention, means (A) for reading data from a medium on which voice / image and date / time information are digitally recorded, means for restoring voice data, means for detecting a voice interruption position, and voice Means (B) for converting the voiced part into phrase units, voice recognition means (C) for converting phrase-based voices into text data, means (D) for additionally creating the date / time information to phrase-based text data, and voice presence The present invention is characterized by having means (E) for performing phrase processing on video data synchronized with the timing of converting the sex portion into phrase units, and means for digitally recording the data on a recording medium.

According to a fifth aspect of the present invention, means (A) for reading data from a medium on which voice / image and date / time information are digitally recorded, means for restoring voice data, means for detecting a voice interruption position, and voice Means (B) for converting the voiced part into phrase units, voice recognition means (C) for converting phrase-based voices into text data, means (D) for additionally creating the date / time information to phrase-based text data, and voice presence It is characterized in that it is recorded by a multimedia editing apparatus having means (E) for performing phrase processing of video data synchronized with the timing at which the sex part is made into a phrase unit, and means for digitally recording it on a recording medium.

According to a sixth aspect of the present invention, means (AA) for selectively selecting and reading data from a medium in which audio / video and date / time information from a plurality of input data are digitally recorded, means for restoring audio data, and audio Means for detecting a discontinuity position, and means for converting the voiced part of the voice into phrase units (B1)
A voice recognition means (C1) for converting the phrase unit voice into text data, and a means (D1) for additionally creating the date and time information to the phrase unit text data, and another source from the medium (AA). ) Means for selectively reading out, means for restoring voice data, means for detecting voice interruption positions, means (B2) for converting the voiced part into phrase units, and voice recognition for converting phrase units into text data. Means (C2) and means (D2) for adding and creating the date and time information to phrase-based text data, and means (F) for alternately arranging the text data created in (D1) and (D2) in time order. And output means (G).

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

The present invention continuously records each person's utterance with a microphone camera in a meeting of a plurality of persons. For example, in the case of a conference of three people, the input of the microphone / camera is simultaneously recorded for each of the three channels (here, the case of two persons will be described). Some of the utterances in the conference may be simultaneous utterances, but basically they are banquet utterances instead of someone, and are composed of a combination of utterance phrase portions of each person. It is to re-edit the recorded speech phrase of each person into a unit, and characterize the speech content.

The present invention is new in that, as shown in the block configuration example of FIG. 1 or 2, time information is added to a plurality of input sources for each unit recording time and recording is performed in the format of FIG. As a conventional example, an example of recording in a VCR is shown in FIGS.
Shown in 1.

FIG. 1 is a diagram showing a configuration example in which time information is added to video and audio of a plurality of digital sources and recorded.
FIG. 2 is a diagram showing a configuration example in which time information is added to video and audio of an analog source and recorded. The difference between FIG. 1 and FIG. 2 is the difference in the input source. In FIG. 1, the input source is a DV format, and there is a plurality of one input source such as CAM coder in which video and audio are digitized. Figure 2 shows S signal, composite, and component (NTSC, PA
L, SECAM) This is a case where there are a plurality of orthodox input sources in which video signals and audio signals are different. Input channel 1, date and time management block 2, shaping section 3,
It is composed of a switch 4, a writing unit 5, and a recording medium 6.

Each input source receives a signal on the input channel 1 and receives the time information from the date / time management block 2.
The format (3-1) of (3) is divided and shaped for each unit time defined in advance. The recording means continuously writes this on the recording medium (3-2). Here, the unit recording time is a unit of several hundred millimeters to 10 seconds. The switch 4 in the portion for writing information of two channels in FIGS. 1 and 2 is for explaining simultaneous writing by time division.

The video signals are DV input signals, S signals,
Digital compression is performed on both composite and component signals. The compression method is the well-known MPEG but Motion JP
Either EG or JPEG2000 continuous or both may be used. Similarly, audio information is digitized or re-digitized, but a relatively wide band of about 192 KHz to 96 KHz is used, but monaural input is basically used, which is much smaller than the amount of video information.

The recording medium 6 in FIG. 3 is a DVD-ROM, a DVD-RW, a DVD-RAM in addition to the HDD pack device.
Including large-capacity optical discs, flash memory, etc. To each unit time information, an "input channel number", a "session number" indicating the number of times of recording on the same medium, and a "sequence number" indicating what number of the unit time are added are added to the "Project management unit". , Which has a directory management function for managing the entire recorded contents of the medium, and is recorded as "raw data".

Incidentally, the start timing of the unit recording time m may be adjusted by dividing a plurality of input channels at the same timing, adjusting the buffer for delaying the writing, and shifting the writing. Here, since a block (not shown) that shifts the input of each input channel according to the number of input channels n to shift the timing of sequence delimitation every m / n is provided, the use efficiency of the entire memory buffer is increased. Is good.

FIG. 4 shows an example of a structure in which the utterance is displayed in characters in synchronism with the image and voice. According to the first and second aspects, while reproducing the recorded data recorded in one channel from the recorded medium, the utterance is converted into text, and the real time of the utterance is added and output. FIG. 4 shows a recording device 6, a data reading unit A, and an audio decoder unit B.
A text data conversion section C, a phrase storage section D, an output I / F section G, and a video decoder section V.

For reproduction, a channel number and a session number are designated from the recording medium, the sequence number is read in the data reading section A, the video data is decoded by the video decoder V, and the sound is decoded by the audio decoder B. , Audio and time information are separated, and video and audio signals are transmitted from the input / output I / F unit G to an external TV or the like. At the same time, the time information for each sequence is received from the audio decoder B, and the timer measurement is started. Then, upon receiving a notification of the detection of the voiced part of the voice signal from the voice decoder B, the start position time of the voiced part is recalculated. This voiced voice unit is called a "phrase". The phrase storage section D temporarily stores the phrase start time. In the text data conversion section C, a voiced sound section is provided (see JP-A-6-318096 or 2001).
A known technology of 166790 is used), and the voice recognition character code is converted and sent to the phrase storage section D. The phrase start time and the phrase number, which are temporarily stored in the phrase storage section D, are added to the voice recognition character code to prepare the output format shown in FIG.

The output of the character code is output from the input / output I / F section G to an external text monitor, visualized from the internal character font on the text monitor and displayed on the screen.

According to the third aspect, the voice recognition character is output to the external display device at the same time as the reproduction of the video / audio. Next, according to the invention of claim 4, information in which a statement is character coded for each phrase,
A method of reconstructing and recording video / audio for each utterance phrase will be described.

FIG. 6 is a diagram showing an example of the configuration of an apparatus for converting a speech pattern into text and synchronizing it with video and audio and re-recording it. FIG. 7 is a diagram showing a format example of the recording medium 7 of FIG. This configuration has a recording medium 6, a recording medium 7, a reading section A, an audio decoder B, a text data conversion section C,
Phrase storage section D, video decoder V, input / output I / F
And a section G. As shown in FIG. 6, the video data is temporarily stored in the phrase processing unit E. Upon receipt of the phrase detection notification from the text data conversion section C, the image data is reconfigured in the format of 7-1 in FIG. 7 as phrase-based video data. At the same time, it becomes a format including voice data from the data conversion section C and a voice character code (text) with a phrase start time from the phrase storage section D. Here, the audio data may be thinned and compressed from the original data to perform a process (not shown) for saving the medium capacity, and similarly, the video data may be thinned and compressed.

The recorded data in this format is recorded in the recording medium 7 as "Project management" in the recording portion as "sound frame (sound phrase) format" and is distinguishable from "live recording". Become.

7-2 shows an example of a recording structure in which a phrase is formed for each sound phrase. This indicates that the duration is different for each phrase, the phrase data length is recorded in a variable format, and the length is different. Further, a phrase exceeding the maximum phrase data length defined in advance is timely divided by the "subsequence number" of 7-1. This divided phrase data does not include the "text" of the voice recognition output, but "NULL" data is padded.

Further, in the date and time information 7-2, in addition to the start time of each phrase, the end time of each phrase or phrase duration information may be recorded at the same time. Alternatively, blank time information from the end of the previous phrase to the start of the current phrase can be recorded in the next phrase.

FIG. 8 shows an example of the configuration of an apparatus for characterizing and displaying the utterances by synchronizing the patterns of a plurality of speakers with video and audio.
FIG. 9 shows an example of the display. This configuration is for the recording medium 6
, A data selection / readout unit AA, an audio decoder B1,
B2, text data conversion units C1 and C2, phrase storage units D1 and D2, phrase rearrangement unit F, and input / output I
/ F section G and video decoders V1 and V2. As shown in FIG. 3, a plurality of “raw recording” data for each channel (FIG. 3) are alternately read for each channel by the “AA” read block. Then, voice decoding for each channel, voice recognition blocks "B1, C1, D1" and "B2,
In the phrase rearranging section F, the character code with phrase time for each channel processed through “C2, D2” is
The data is rearranged in time order, the channel number is added, and the output format shown in FIG. 9 is prepared. The output of the character code is output from the input / output I / F section G to an external text monitor, visualized from the internal character font on the text monitor, and displayed on the screen.

Here, the rearrangement of the phrase codes in the phrase rearranging section F can be performed in an output form in which a plurality of phrases are connected by adding the determination of the inter-phrase time of the same channel. This is in consideration of the readability when visualizing the phrase formation and sentence composition for voice recognition. The sentence composition phrase time is set to about 10 times the silent detection time for voiced part determination. To be done.

[0036]

As is apparent from the above description, according to the present invention, the speakers from a plurality of input sources can be independently recorded with a simple structure.

Further, according to the present invention, it becomes possible to obtain a utterance record with the utterance time from the voice of the speaker of an independent input source.

Further, according to the present invention, a utterance can be obtained from the voice of a speaker of an independent input source with a utterance time, video and audio data can be compressed and re-recorded, and a recording medium can be saved. The second use of the utterance will be possible. In addition, the voices of speakers from multiple input sources can be batch processed with a simple configuration.
It can be applied to multi-source input processing.

Further, according to the present invention, the utterances of a plurality of speakers can be displayed in text in the order of utterances, and the utterances can be secondarily used.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration example of an apparatus for adding time information to video / audio of a plurality of digital sources and recording the information.

FIG. 2 is a diagram showing a configuration example of an apparatus that adds time information to video / audio of a plurality of analog sources and records the information.

FIG. 3 is a diagram showing a format example of a medium in which time information is added to video / audio of a plurality of sources and recorded.

FIG. 4 is a diagram showing an example of the configuration of an apparatus for characterizing and displaying a message by synchronizing the pattern of the message with video / audio.

FIG. 5 is a diagram showing a display example of an apparatus for characterizing and displaying a message in synchronization with the pattern of the message in synchronization with video / audio.

FIG. 6 is a diagram showing a configuration example of an apparatus for converting a statement into a character in synchronization with video / audio and re-recording the statement.

FIG. 7 is a format example of a medium for recording an example of the configuration of an apparatus for converting an utterance into characters by synchronizing the utterance pattern with video / audio and re-recording.

FIG. 8 is a diagram showing an example of the configuration of an apparatus that characterizes and displays a message by synchronizing the patterns of a plurality of speakers with video and audio.

FIG. 9 is a diagram showing a display example of an apparatus which characterizes and displays a message by synchronizing patterns of a plurality of speakers with video and audio.

FIG. 10 is a diagram showing a conventional example in which a plurality of video / audio sources are recorded in two VCRs.

FIG. 11 is a diagram showing an example of a recording format of a commercial VCR tape as a conventional example.

[Explanation of symbols]

1 input channel 2 Date management block 3 Orthopedic department 4 switches 5 writing section 6, 7 recording medium A data reading section B audio decoder C text data conversion section D phrase storage E phrase processing section F phrase rearrangement section G Input / output I / F section H reconstruction unit V video decoder

─────────────────────────────────────────────────── ─── Continued Front Page (51) Int.Cl. ⁷ Identification Code FI Theme Coat (Reference) H04N 5/781 510J 510L

Claims (6)

[Claims] 1. A means for managing date and time and a plurality of input channel means for simultaneously inputting a plurality of digital data of video synchronized with audio, and means for digitally storing digital data and date and time information of each channel in a recording medium. A multimedia recording device having. 2. A means for managing date and time and a plurality of input channel means for simultaneously inputting a plurality of digital data of video synchronized with audio, and means for digitally storing digital data and date / time information of each channel in a recording medium. A recording medium recorded by a multimedia recording device having the above. 3. A means (A) for reading out data from a medium in which audio / video and date / time information is digitally recorded, a means for restoring audio data, a means for detecting an audio discontinuity position, and a voiced voice unit being phraseized as a unit of phrase. A voice recognition means (C) for converting phrase unit voice into text data, a means (D) for adding and creating the date and time information to the phrase unit text data, and a means for restoring video data. (V)
And a means (G) for outputting the multimedia reproduction apparatus. 4. A means (A) for reading out data from a medium on which voice / image and date / time information are digitally recorded, a means for restoring voice data, a means for detecting a voice interruption position, and a voice voiced portion in phrase unit. Means (B), voice recognition means (C) for converting phrase-based voices into text data, means (D) for additionally creating the date and time information to phrase-based text data, and voice-specific parts for phrase-based conversion A multimedia editing apparatus comprising: means (E) for performing phrase processing of video data synchronized with the timing for performing the digital recording; and means for digitally recording the data on a recording medium. 5. A means (A) for reading out data from a medium in which voice / image and date / time information are digitally recorded, a means for restoring voice data, a means for detecting a voice interruption position, and a voice voice section in a phrase unit. Means (B), voice recognition means (C) for converting phrase-based voices into text data, means (D) for additionally creating the date and time information to phrase-based text data, and voice-specific parts for phrase-based conversion A recording medium recorded by a multimedia editing device having means (E) for performing phrase processing of video data synchronized with the timing for performing the digital recording on the recording medium. 6. A means (AA) for selectively reading data from a medium in which audio / video and date and time information from a plurality of input data are digitally recorded, a means for restoring audio data, and a means for detecting an audio interruption position. A means (B1) for converting the voiced part into phrase units; a voice recognition means (C1) for converting phrase units into text data; and a means (D1) for additionally creating the date / time information to the phrase-based text data. Means for selectively reading another source from the medium by means (AA), means for restoring voice data, means for detecting a voice discontinuity position, means for phraseizing the voiced sound portion into phrases (B2), voice recognition means (C2) for converting phrase unit voice into text data, and the date information is added to the phrase unit text data. (D1), a means (F) for alternately arranging the text data created in (D1) and (D2) in chronological order, and an output means (G) apparatus.