JP5912729B2 - Speech recognition apparatus, speech recognition program, and speech recognition method - Google Patents

Description

  The present invention relates to a speech recognition apparatus, a speech recognition program, and a speech recognition method for recognizing speech included in multimedia information.

  2. Description of the Related Art Various types of multimedia information have been widely provided by distribution of moving images and sounds by live broadcasting, on-demand distribution by streaming of previously recorded and recorded moving images and sounds, and the like.

  Here, a comment distribution system is proposed in which when a user who listens to multimedia information inputs a comment on the multimedia information while listening, the comment is presented to other users who listen to the multimedia information. (See Patent Document 1).

  On the other hand, a technique has been proposed in which speech recognition is performed in units of words using candidate words prepared in advance and their appearance probabilities (see Non-Patent Document 1). Furthermore, a technique for improving the accuracy of speech recognition by analyzing temporal correspondence between speech and text transcribed from the speech by dictation has been proposed (see Patent Document 2).

Japanese Patent No. 4263218 Japanese Patent No. 4758919

Akinobu Lee and Tatsuya Kawahara, Recent Development of Open-Source Speech Recognition Engine Julius, Proceedings: APSIPA ASC 2009: Asia-Pacific Signal and Information Processing Association, 2009 Annual Summit and Conference, pp.131-137, October 4, 2009 Published by http://hdl.handle.net/2115/39653

  In the current situation where a large amount of multimedia information is provided, there is an increasing demand for subtitles for videos included in the multimedia information, provision of multimedia information summary texts, and retrieval of multimedia information texts. . Therefore, there is a strong demand for making it possible to more appropriately convert the voice included in the multimedia information into text.

  On the other hand, since words appearing in speech change according to topics, trends in the times, preferences of speakers and listeners, etc., a dictation technique that can immediately respond to such changes is required.

  The present invention is intended to solve such a problem, and uses a comment attached to multimedia information to appropriately recognize a voice included in the multimedia information, a voice recognition program, It is another object of the present invention to provide a speech recognition method.

In order to achieve the above object, a speech recognition apparatus according to the first aspect of the present invention provides:
An accumulator for accumulating comments input by the user while listening to audio generated by playing multimedia information;
An extraction unit that extracts words that appear in a sentence set including the accumulated comments and candidate words including co-occurrence words of the word in the sentence set;
A voice recognition unit that recognizes a voice generated by reproducing the multimedia information based on the extracted candidate word;
It is characterized by that.

In the speech recognition apparatus according to the first aspect,
The sentence set may include a sentence that appears in a document viewed by a user who has listened to the multimedia information.

In the speech recognition apparatus according to the first aspect,
The extraction unit calculates the likelihood of appearance of each of the candidate words,
The speech recognition unit recognizes speech based on the degree of coincidence between the phoneme recognized from the speech and the phoneme representing the candidate word and the appearance likelihood of the candidate word.
It is also good.

In the speech recognition apparatus according to the first aspect,
Of the candidate words, words appearing in the comment are associated with an input time point when the comment is input,
For the candidate word associated with the input time point, the speech recognition unit obtains a degree of match between the input time point associated with the candidate word and the pronunciation time point when the phoneme is emitted, Voice recognition based on the degree of match
It is also good.

In the speech recognition apparatus according to the first aspect,
The input time point and the sound generation time point are expressed by a reproduction time after the reproduction of the multimedia information is started.
It is also good.

In the speech recognition apparatus according to the first aspect,
The degree of match is determined based on a difference between the input time point and the sound generation time point, and a difference between a time point when the multimedia information can be played back and a time point when the user starts playing the multimedia information.
It is also good.

A speech recognition program according to the second aspect of the present invention is:
Computer
An accumulator for accumulating comments input by the user while listening to audio generated by playing multimedia information;
An extraction unit that extracts words that appear in a sentence set including the accumulated comments and candidate words including co-occurrence words of the word in the sentence set;
Based on the extracted candidate words, function as a speech recognition unit that recognizes speech generated by playing the multimedia information,
It is characterized by that.

Furthermore, the speech recognition method according to the third aspect of the present invention provides:
A method performed by a speech recognition apparatus including an accumulation unit, an extraction unit, and a speech recognition unit,
An accumulating step in which the accumulating unit accumulates a comment input by the user while listening to a sound uttered by reproduction of multimedia information;
An extraction step in which the extraction unit extracts a word that appears in a sentence set including the accumulated comments and a candidate word including a co-occurrence word of the word in the sentence set;
The speech recognition unit has a speech recognition step for recognizing speech generated by reproducing the multimedia information based on the extracted candidate words.
It is characterized by that.

  According to the voice recognition device, the voice recognition program, and the voice recognition method according to the present invention, the voice included in the multimedia information can be appropriately recognized using the comment attached to the multimedia information.

It is a system configuration figure showing an example of 1 composition of a voice recognition system. It is a hardware block diagram showing an example of the speech recognition apparatus which concerns on the Example of this invention. It is a flowchart showing an example of the live broadcast process which a speech recognition apparatus performs. FIG. 3 is a functional block diagram illustrating an example of functions of the voice recognition device according to the first embodiment. It is a figure showing an example of the broadcast table which a voice recognition apparatus memorizes. It is a figure showing an example of the comment table which a speech recognition apparatus memorizes. It is a figure showing an example of the viewing-and-listening screen which the terminal device concerning Example 1 displays. It is a flowchart showing an example of the rebroadcast process which a speech recognition apparatus performs. 6 is a flowchart illustrating an example of a summary generation process executed by the speech recognition apparatus according to the first embodiment. It is a figure showing an example of the reference table which a speech recognition apparatus memorize | stores. It is a figure showing an example of the sentence set table which a speech recognition device memorizes. It is a figure showing an example of the co-occurrence table which the speech recognition apparatus concerning Example 1 memorizes. It is a figure showing an example of the candidate word table which a speech recognition apparatus memorizes. It is a figure showing an example of the coincidence degree curve represented by the information which a voice recognition device memorizes. It is a flowchart showing an example of the sentence set production | generation process which a speech recognition apparatus performs. It is a flowchart showing an example of the candidate word extraction process which a speech recognition apparatus performs. It is a flowchart showing an example of the continuous speech recognition process which a speech recognition apparatus performs. 10 is a flowchart illustrating an example of a summary generation process executed by the speech recognition apparatus according to the second embodiment. It is a functional block diagram showing an example of the function which the speech recognition apparatus concerning Example 2 has. It is a figure showing an example of the co-occurrence table which the speech recognition apparatus concerning Example 2 memorize | stores. It is a figure showing an example of the viewing-and-listening screen which the terminal device concerning Example 3 displays.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

<Example 1>
A speech recognition apparatus 100 according to Embodiment 1 of the present invention constitutes a speech recognition system 1 as shown in FIG.

  In addition to the speech recognition device 100, the speech recognition system 1 includes, for example, a computer communication network 10 such as the Internet (hereinafter simply referred to as the communication network 10), and terminal devices 20, 30, and 40 connected to the communication network 10. , Is composed.

  The terminal devices 20 to 40 are each configured by a personal computer including a display unit such as an LCD (Liquid Crystal Display), an audio output unit such as a speaker, and an input unit such as a keyboard and a mouse.

  In addition, the terminal device 20 is connected to an imaging device 21 such as a web camera and a sound collection device 22 such as a microphone.

  The voice recognition device 100 receives multimedia information representing the moving image shot by the imaging device 21 and the voice collected by the voice collection device 22 from the terminal device 20, and distributes the received multimedia information to the terminal devices 20 to 40. To do. Thereby, the moving image photographed by the imaging device 21 and the sound collected by the sound collecting device 22 are broadcast as the video and sound of the program.

  Here, the voice recognition device 100 will be described as broadcasting a program in which the user of the terminal device 20 appears to the terminal devices 20 and 30 within a predetermined time from the recording of the program (hereinafter referred to as live broadcasting). In addition, the user of the terminal device 20 performs while viewing the broadcasted program.

  Further, here, the speech recognition apparatus 100 is described as broadcasting a live broadcast program (hereinafter referred to as a live broadcast program) to the terminal device 40 (hereinafter referred to as rebroadcast) after a predetermined time has elapsed since the recording of the program. .

Next, the hardware configuration of the speech recognition apparatus 100 will be described with reference to FIG.
The speech recognition apparatus 100 includes a server machine as shown in FIG. 2, and includes a central processing unit (CPU) 101, a read only memory (ROM) 102, a random access memory (RAM) 103, a hard disk 104, a media controller 105, a LAN. A card (Local Area Network) 106, a video card 107, an LCD (Liquid Crystal Display) 108, a keyboard 100i, a speaker 110, and a touch pad 111 are included.

  The CPU 101 performs overall control of the speech recognition apparatus 100 by executing a program according to a program stored in the ROM 102 or the hard disk 104. The RAM 103 is a work memory that temporarily stores data to be processed when the CPU 101 executes a program.

  The hard disk 104 is a storage unit that stores a table storing various data. Note that the speech recognition apparatus 100 may include a flash memory instead of the hard disk 104.

  The media controller 105 reads various data and programs from recording media including flash memory, CD (Compact Disc), DVD (Digital Versatile Disc), and Blu-ray Disc (registered trademark).

  The LAN card 106 transmits and receives data to and from the terminal devices 20 to 40 connected via the communication network 10. The keyboard 100i and the touch pad 111 input signals according to user operations.

  The video card 107 draws (that is, renders) an image based on the digital signal output from the CPU 101, and outputs an image signal representing the drawn image. The LCD 108 displays an image according to the image signal output from the video card 107. The voice recognition apparatus 100 may include a PDP (Plasma Display Panel) or an EL (Electroluminescence) display instead of the LCD 108. The speaker 110 outputs sound based on the signal output from the CPU 101.

Next, functions of the speech recognition apparatus 100 will be described.
The CPU 101 functions as the input unit 120, the storage unit 130, and the output unit 140 illustrated in FIG. 4 by executing the live broadcast processing illustrated in FIG. Further, the CPU 101 functions as the storage unit 190 in cooperation with the hard disk 104 shown in FIG.

  The input unit 120 illustrated in FIG. 4 inputs various types of information received by the LAN card 106 illustrated in FIG. The storage unit 130 stores various types of information input by the input unit 120 in the storage unit 190. The output unit 140 outputs various information input from the input unit 120 to the LAN card 106 by designating a distribution destination. The storage unit 190 stores various types of information stored by the storage unit 130.

Next, various information stored in the storage unit 190 will be described.
The storage unit 190 stores a broadcast table shown in FIG. 5 in which bibliographic items of broadcast programs are stored. The broadcast table corresponds to the broadcast ID for identifying the broadcast of the program, the broadcast start date and time of the program, the shift time of the broadcast, and the path of multimedia information representing the video and audio broadcast in the program. Multiple attached data are saved. Note that the broadcast start date and time of the program refers to the date and time when the broadcast of the program was started. The broadcast shift time is a value “0” when the broadcast is a live broadcast, and is a value obtained by subtracting the start date and time of the live broadcast from the start date and time of the rebroadcast when the broadcast is a rebroadcast.

  In addition, the storage unit 190 stores a comment table shown in FIG. 6 in which comments about the moving image or sound of the program are stored. The comment table includes data in which a broadcast ID of a program, a comment ID that identifies a comment on the program, an input time point of the comment, the comment, and a user ID that identifies the commented user are associated with each other. Multiple saved. The input time is represented by the elapsed time from the start of the program broadcast.

  Next, operations of the CPU 101 performed by the input unit 120, the storage unit 130, and the output unit 140 illustrated in FIG. 4 will be described.

  The user performs an operation for instructing the start of live broadcast (hereinafter referred to as a live broadcast start instruction operation) on the keyboard 109 of the speech recognition apparatus 100. Next, the user instructs the keyboard 109 to specify the date and time when the broadcast is scheduled to start (hereinafter referred to as the broadcast start scheduled date and time) and the date and time when the broadcast is scheduled to end (hereinafter referred to as the scheduled broadcast end date and time). Perform the operation.

  When an operation signal corresponding to a live broadcast start instruction operation is input from the keyboard 109, the CPU 101 starts execution of the live broadcast process shown in FIG.

  When the execution of the live broadcast process is started, the input unit 120 generates a broadcast ID, and acquires a scheduled broadcast start date and a scheduled broadcast end date and time designated by the user operation based on an operation signal input from the keyboard 109. (Step S01).

  Next, the storage unit 130 refers to, for example, a system date and time managed by an OS (Operating System), and determines whether or not the referred system date and time is a date and time when the scheduled broadcast start date and time has passed (step S02). . At this time, if the storage unit 130 determines that the scheduled broadcast start date and time has not elapsed (step S02; No), the process proceeds to step S02 after sleeping for a predetermined time.

  In step S02, when the storage unit 130 determines that the scheduled broadcast start date / time has elapsed (step S02; Yes), the stored system date / time is set as the broadcast start date / time. Since the storage unit 130 is a live broadcast, the shift time of the program is set to a value “0”. Further, the storage unit 130 generates an electronic file path in which multimedia information representing the moving image and audio of the program is stored, and creates an electronic file in the generated path. Next, the storage unit 130 associates the broadcast ID, the broadcast start date and time, the shift time, and the path, and additionally stores them in the broadcast table of FIG. 5 (step S03).

  Next, the storage unit 130 starts a software timer and starts counting the elapsed time from the start of program broadcasting (step S04).

  Here, since the scheduled broadcast start date and time has passed, the user of the terminal device 20 performs an operation of causing the photographing device 21 connected to the terminal device 20 to start photographing and causing the sound collecting device 22 to start collecting sound. A description will be given assuming that the process is performed on the terminal device 20.

  The terminal device 20 starts photographing by the photographing device 21 and voice collection by the voice collecting device 22 in accordance with the operation. Next, for example, the terminal device 20 starts to input data (hereinafter referred to as “moving image data”) representing a moving image of the appearance of the performer from the shooting device 21. In addition, the terminal device 20 starts to input an electric signal (hereinafter referred to as an audio signal) representing an audio such as a speech of a performer from the audio input device 22, for example. Thereafter, the terminal device 20 generates audio data based on the input audio signal, and the generated audio data and the moving image data input from the imaging device 21 are associated with the data input date / time and the generation date / time. The transmission of the media information to the voice recognition device 100 is started.

  Next, the input unit 120 inputs the multimedia information received by the LAN card 106 from the terminal device 20 from the LAN card 106 shown in FIG. 2 (step S05).

  Next, the storage unit 130 additionally stores the input multimedia information in the electronic file in the path described above (step S06).

  Thereafter, the output unit 140 outputs the input multimedia information to the LAN card 106 shown in FIG. 2 with the terminal devices 20 and 30 as destinations (step S07). Thereafter, the LAN card 106 distributes the multimedia information to the terminal devices 20 and 30 (that is, live broadcast).

  Here, when receiving the multimedia information from the voice recognition device 100, the terminal devices 20 and 30 display a viewing screen shown in FIG. 7 that displays a moving image represented by the multimedia information. Next, the terminal devices 20 and 30 display the moving image in which the multimedia information is reproduced in the moving image display area AM of the viewing screen, and output the reproduced sound from the audio output device.

  Here, a description will be given assuming that the user of the terminal device 20 makes a statement saying “Because the metropolitan government is confused” while facing the front of the photographing device 21. For this reason, on the viewing screen shown in FIG. 2, a moving image obtained by shooting the state in which the user of the terminal device 20 speaks from the front is displayed, and the voice “Because the metropolitan administration is confused” is output from the terminal devices 20 and 30. .

  Thereafter, the user of the terminal device 20 and the terminal device 30 who has watched the program may or may not perform an operation for inputting a comment of the viewed program on the terminal device 30. At this time, when the user performs the operation on the terminal device 30, the terminal device 30 inputs a comment, and transmits the comment information indicating the input comment and the user ID of the commented user to the voice recognition device 100. To do.

  After step S07 shown in FIG. 3 is executed, the input unit 120 inputs the multimedia information by executing the same process as step S05 (step S08).

  Thereafter, the input unit 120 determines whether or not the LAN card 106 has received comment information based on a signal output from the LAN card 106 illustrated in FIG. 2 (step S09).

  At this time, if the input unit 120 determines that the LAN card 106 has not received the comment information (step S09; No), the input unit 120 performs the same processing as step S06 and step S07, thereby saving and outputting the comment information. It performs (step S10 and step S11).

  In contrast, when the input unit 120 determines that the LAN card 106 has received the comment information (step S09; Yes), the input unit 120 inputs the comment information received by the LAN card 106 and the user ID from the LAN card 106. (Step S12).

  After that, the storage unit 130 refers to the software timer and acquires the elapsed time from the start date and time of live broadcasting (step S13). Next, the storage unit 130 sets the acquired elapsed time as a comment input time (step S14). Thereafter, the storage unit 130 generates a comment ID of the comment represented by the comment information.

  Next, the storage unit 130 associates the broadcast ID of the program, the input time and comment ID of the comment for the program, the comment, and the user ID of the user who issued the comment, as shown in FIG. It is additionally stored in the comment table (step S15).

  Thereafter, the output unit 140 outputs the input comment information to the LAN card 106 illustrated in FIG. 2 with the terminal devices 20 and 30 as destinations (step S16). Thereafter, the LAN card 106 distributes the comment information to the terminal devices 20 and 30.

  When receiving the comment information from the voice recognition device 100, the terminal devices 20 and 30 display the comment represented by the comment information in the comment display area AC of the viewing screen shown in FIG.

  Next, the storage unit 130 synthesizes the comment represented by the comment information input at step S12 with the moving image represented by the multimedia information input at step S08 (step S17).

  Thereafter, the storage unit 130 additionally stores the multimedia information representing the moving image with the combined comment in the file in the above-described path (step S18).

  Next, the output unit 140 outputs the multimedia information combined with the comments to the LAN card 106 shown in FIG. 2 with the terminal devices 20 and 30 as destinations (step S19). Thereafter, the LAN card 106 distributes the multimedia information to the terminal devices 20 and 30.

  When the terminal devices 20 and 30 receive the multimedia information from the voice recognition device 100, the terminal devices 20 and 30 reproduce the multimedia information, and display the moving image in which the comment is synthesized in the moving image display area AM of the viewing screen shown in FIG.

  Here, it is assumed that a viewer who uses the terminal device 30 listens to the output voice “Because the metropolitan government is confused”, and inputs a comment “It is too confusing” to the voice to the terminal device 30. . Further, the viewer will be described assuming that the video of the performer displayed on the viewing screen is visually recognized and the comment “Ichiro Sato Kita!” Referring to the name of the performer is input to the terminal device 30. Therefore, in the comment display area AC of the viewing screen shown in FIG. 7, the comments “I'm too confused” and “Ichiro Sato!” Are displayed. In the moving image display area AM, a moving image in which the comments “I'm too confused” and “Ichiro Sato!” Is displayed on the front image of the performer is displayed.

  After step S11 or step S19 is executed, the input unit 120 refers to the system date and time to determine whether or not the referenced system date is the date and time when the scheduled live broadcast end date acquired in step S01 has passed ( Step S20). At this time, if the input unit 120 determines that the scheduled live broadcast end date has not elapsed (step S20; No), the input unit 120 repeats the above-described processing from step S08.

  In step S20, when the input unit 120 determines that the scheduled live broadcast end date has passed (step S20; Yes), the input of the live broadcast process ends.

  Next, the operation of the CPU 101 will be described by taking as an example a case where the voice recognition device 100 rebroadcasts a program that has already been broadcast live and the user of the terminal device 40 views the program.

  Here, the user of the terminal device 40 performs an operation for transmitting to the voice recognition device 100 a request for requesting rebroadcast of a live broadcast program (hereinafter referred to as a rebroadcast request) after a predetermined time has elapsed since the start of live broadcast. To 40. The terminal device 40 transmits a rebroadcast request to the voice recognition device 100 according to the operation.

  When the LAN card 106 shown in FIG. 2 receives the rebroadcast request, the CPU 101 starts executing the rebroadcast process shown in FIG.

  First, the input unit 120 generates a broadcast ID and inputs the received rebroadcast request from the LAN card 106. Next, the input unit 120 acquires from the rebroadcast request the broadcast ID of the live broadcast program for which rebroadcast is requested, and the date and time for requesting the start of rebroadcast (hereinafter referred to as rebroadcast request date and time) (step S31).

  Next, the storage unit 130 refers to the system date and time, and determines whether or not the referenced system date and time is the date and time when the live broadcast start request date and time has passed (step S32). At this time, if it is determined that the rebroadcast start request date has not elapsed (step S32; No), the storage unit 130 waits for a predetermined time and then repeats the process of step S32.

  In step S32, when determining that the rebroadcast start request date / time has passed (step S32; Yes), the storage unit 130 refers to the system date / time and sets the referred system date / time as the rebroadcast start date / time. Further, the storage unit 130 searches the broadcast table shown in FIG. 5 for the broadcast start date and time and the path associated with the broadcast ID of the live broadcast program for which rebroadcast is requested. Thereafter, the storage unit 130 calculates a difference between the broadcast start date and time of the rebroadcast and the broadcast start date and time of the live broadcast, and sets the calculated difference as the shift time. Next, the storage unit 130 associates the broadcast ID of the rebroadcast, the broadcast start date and time of the rebroadcast, the shift time of the rebroadcast, and the path of the rebroadcast live program, as shown in FIG. It is additionally stored in the broadcast table (step S33).

  Next, the preservation | save part 130 starts the measurement of the elapsed time from a rebroadcast start date by performing the process similar to step S04 (step S34).

  Next, the input unit 120 reads multimedia information of a predetermined size from the electronic file in the above path (step S35).

  Thereafter, the output unit 140 outputs the read multimedia information to the LAN card 106 shown in FIG. 3 with the terminal device 40 as the destination (step S37). Thereafter, the LAN card 106 transmits multimedia information to the terminal device 40. The terminal device 40 reproduces the received multimedia information (so-called time-shifted reproduction), displays a moving image in which a comment input by the user of the terminal device 30 is synthesized, and outputs a sound.

  Thereafter, the user of the terminal device 40 may or may not perform an operation of viewing the rebroadcast program and inputting a comment on the program on the terminal device 40.

  Next, the input unit 120 executes the same processing as step S35 and reads the multimedia information (step S38).

  Thereafter, the input unit 120 determines whether or not the LAN card 106 has received the comment information by executing the same process as in step S09 illustrated in FIG. 3 (step S39).

  At this time, if the input unit 120 determines that the LAN card 106 has not received the comment information (step S39; No), the input unit 120 executes the same process as the process of step S37, thereby reading the multi data read in step S38. Media information is output (step S41).

  In step S39, when the input unit 120 determines that the LAN card 106 has received the comment information (step S39; Yes), the input unit 120 executes processing similar to the processing from step S12 to step S17 in FIG. 3 (from step S42). Step S47). As a result, multimedia information in which the comment represented by the comment information input at step S42 is combined with the moving image represented by the multimedia information read at step S38 is generated.

  Next, the storage unit 130 rewrites the multimedia information read in step S38 among the multimedia information stored in the electronic file in the above-described path with the multimedia information generated in step S47 ( Step S48).

  Thereafter, the output unit 140 executes the same process as step S19 shown in FIG. 3 (step S49). Thereby, the multimedia information representing the moving image in which the comment input by the user of the terminal device 40 is combined is transmitted to the terminal device 40.

  After the process of step S41 or step S49 is executed, the input unit 120 moves the position where multimedia information is read from the electronic file in the above-described path (hereinafter referred to as a read position) by the size of the read multimedia information. Shift to the side. Next, the input unit 120 determines whether or not the reading position is an end of file (EOF) that is the last of the electronic file (step S50). At this time, if the input unit 120 determines that the reading position is not EOF (step S50; No), the above processing is repeated from step S38.

  In step S50, when the input unit 120 determines that the reading position is EOF (step S50; Yes), the execution of the rebroadcast process is terminated.

  The CPU 101 of the speech recognition apparatus 100 generates a text summarizing the content of the remarks in the program as a search key for the broadcast program or a caption attached to a moving image broadcast in the program, as shown in FIG. Execute. Thereby, the CPU 101 functions as the extraction unit 150 and the voice recognition unit 160 in addition to the above-described input unit 120, storage unit 130, and output unit 140 shown in FIG. Further, the CPU 101 functions as the storage unit 190 in cooperation with the hard disk 104 as described above.

  The extraction unit 150 extracts words (hereinafter referred to as “candidate words”) that are candidates for the words representing the voice uttered in the program from comments and the like stored in the storage unit 190. The voice recognition unit 160 recognizes a voice uttered by playing multimedia information based on the extracted candidate words.

Next, various information used for the summary generation process will be described.
The storage unit 190 stores a reference table shown in FIG. 10 in which URLs of documents referred to by users who comment on programs are stored. The reference table associates the user ID of the user, the URL (Uniform Resource Locator) of the document referred to by the user, and the date and time (hereinafter referred to as reference date and time) when the user referred to the document at the URL. Multiple saved data are stored.

  The document referred to by the user includes, for example, a web page or a blog on which news, encyclopedias, and dictionary contents are posted. Further, the speech recognition apparatus 100 functions as a document server, and receives a document transmission request, a URL of a document requesting transmission, and a user ID of a user requesting transmission from each of the terminal devices 20 to 40. . The speech recognition apparatus 100 returns the document requested to be transmitted, and associates the user ID, the request reply date and time (that is, the user reference date and time), and the document URL with reference to FIG. Accumulate in the table.

  Further, the storage unit 190 stores a sentence set table shown in FIG. 11 in which a sentence set whose elements are sentences related to a program is stored. Here, the sentence related to the program includes a sentence constituting the comment of the input program (hereinafter referred to as an input sentence) and a sentence (hereinafter referred to as a reference sentence) posted in a document referred to by the user who has commented on the program. Including.

  In the sentence set table, when a sentence related to a program is an input sentence, the sentence ID for identifying the sentence, the sentence, the type of the sentence, the input time of the sentence, and the broadcast start of the program A plurality of data in which the date / time shift time (hereinafter referred to as the shift time corresponding to the sentence) is associated with each other are stored.

  In the sentence set table, when a sentence related to a program included in the sentence set is a reference sentence, the sentence ID for identifying the sentence, the sentence, the type of the sentence, and the search for the sentence are included. A plurality of data in which the input time of the used comment is associated with the shift time corresponding to the sentence is stored.

  Further, the storage unit 190 stores a co-occurrence word table shown in FIG. 12 in which words that may be included in comments and documents and co-occurrence words that may be used with the words in comments and documents are stored. doing. In the co-occurrence word table, how likely is the word, the co-occurrence word for the word, and the word and the co-occurrence word to be used together in a comment or document (ie, co-occurring) A plurality of pieces of data in which the likelihood (hereinafter referred to as a co-occurrence likelihood) is represented is stored.

  Further, the storage unit 190 stores a candidate word table shown in FIG. 13 in which candidate words are stored. In this embodiment, the speech recognition apparatus 100 refers to a word included in an input sentence (hereinafter referred to as an input word) as a word candidate representing a voice uttered in a program, and the user refers to the time when the input sentence is input. Words included in the reference sentence (hereinafter referred to as reference words) and their co-occurrence words (hereinafter referred to as input co-occurrence words and reference co-occurrence words, respectively) are used.

  Therefore, in the candidate word table, when the candidate word is an input word, the candidate word ID for identifying the input word, the input word, and the input time point of the input sentence including the input word (hereinafter, the input word) An input time corresponding to a word), a shift time corresponding to a sentence including the input word (hereinafter referred to as a shift time corresponding to the input word), and an appearance likelihood of the input word are stored in association with each other. Is done. The appearance likelihood is a value representing the likelihood that the candidate word appears in the utterance in the program under the condition that the comment used for extracting the candidate word is input.

  Further, in the candidate word table, when the candidate word is a reference word, the candidate word ID of the reference word, the reference word, and the input time point of the comment used for searching the document including the reference word (hereinafter, The input time corresponding to the reference word), the shift time corresponding to the sentence including the reference word (hereinafter referred to as the shift time corresponding to the reference word), and the appearance likelihood of the reference word Saved.

  Further, in the candidate word table, when the candidate word is an input co-occurrence word, the input that is assumed to be used together with the candidate co-occurrence word ID, the input co-occurrence word, and the input co-occurrence word The input time of the word (hereinafter referred to as the input time corresponding to the input co-occurrence word), the shift time corresponding to the sentence including the input word (hereinafter referred to as the shift time corresponding to the input co-occurrence word), the The appearance likelihood of the input co-occurrence word is stored in association with each other.

  Furthermore, in the candidate word table, when the candidate word is a reference co-occurrence word, it is estimated that the candidate word ID, the reference co-occurrence word, and the reference co-occurrence word are used together with the reference co-occurrence word. An input time corresponding to the reference word (hereinafter referred to as an input time corresponding to the reference co-occurrence word) and a shift time corresponding to a sentence including the reference word (hereinafter referred to as a shift time corresponding to the reference co-occurrence word); The appearance likelihood of the reference co-occurrence word is stored in association with each other.

  In addition, the storage unit 190 stores an acoustic model, a word dictionary, and a language model that are used to recognize the sound of the program. The acoustic model represents a frequency pattern of phonemes and syllables, and is used to decompose a program sound into an array of phonemes or syllables (hereinafter referred to as phonemes) (hereinafter referred to as a phoneme sequence). The word dictionary is a dictionary in which a plurality of words and phoneme sequences representing pronunciations of the words are associated with each other. The language model is a model that defines a chain of a plurality of words, and is a bigram model that defines a chain of two words or a trigram model that defines a chain of three words. It may be an N-gram model that defines a chain of words.

  In addition, the accumulating unit 190 is a degree-of-match data representing a degree of coincidence that indicates to what extent a voice that is pronounced at a certain pronunciation point matches a voice that is the subject of a comment that is input at a certain input point. Is remembered. The degree-of-match data represents a degree-of-match curve representing how the degree of match changes with a change in the difference obtained by subtracting the pronunciation time from the input time (hereinafter referred to as time difference).

  The matching degree curve stored by the storage unit 190 includes a live broadcast matching degree curve and a rebroadcast matching degree curve. The live broadcast match level curve represents the degree of match between the sound of a program broadcast live and the sound of a comment input during the program broadcast. The rebroadcast match curve represents the degree of match between the sound of the rebroadcast program and the sound of the comment input during the rebroadcast of the program.

  The points on the rebroadcast match score curve have a greater match score than the points on the live broadcast match score curve within a time difference between the predetermined value “−TD1” and “+ TD2”. A viewer who has already watched a program by live broadcasting or a viewer who has repeatedly viewed the same program by rebroadcast knows in advance the sound broadcast by the program. For this reason, these viewers tend to input comments at a time closer to the time of pronunciation of the speech to be commented than viewers viewing the program for the first time in a live broadcast.

  In addition, the live broadcast match degree curve has a peak when the time difference is “TP”, and attenuates as the time difference moves away from “TP”. This is because in the case of live broadcasting, a comment is often input to the sound after listening to the sound of the performer. However, since the performer may speak in response to the input comment, the time difference is not necessarily positive (that is, the comment input time is later than the pronunciation time).

  Further, the rebroadcast match degree curve has a peak when the time difference is “0”, and attenuates as the distance from the time difference “0” increases. This is because, as described above, viewers who have already watched a program on a live broadcast often input a comment at the same time as the sound of the voice to be commented.

  Next, operations of the CPU 101 performed by the input unit 120, the storage unit 130, the output unit 140, the extraction unit 150, and the voice recognition unit 160 illustrated in FIG. 4 will be described.

  When the broadcast ends, the user of the speech recognition apparatus 100 performs an operation for instructing to generate a text summarizing the contents of the audio broadcast in the program (hereinafter referred to as a summary generation instruction operation), and a program for generating a summary. An operation for designating a path of multimedia information (hereinafter referred to as a path designating operation) is performed on the keyboard 109 shown in FIG.

  When the CPU 101 of the speech recognition apparatus 100 inputs a signal corresponding to the summary generation instruction operation from the keyboard 109, the CPU 101 starts executing the summary generation process shown in FIG.

  First, the input unit 120 receives a signal output from the keyboard 109, and specifies a path specified by a path specifying operation (hereinafter referred to as a specified path) based on the input signal (step S61).

  Next, the extraction unit 150 executes a sentence set generation process shown in FIG. 15 that generates a sentence set having a sentence related to the program represented by the multimedia information in the path as an element (step S62).

  When the sentence set generation process is started, the extraction unit 150 searches all broadcast IDs associated with the designated path from the broadcast table shown in FIG. 5 (step S71).

  Next, for each searched broadcast ID (hereinafter referred to as a search broadcast ID), the extraction unit 150 shows a comment associated with the search broadcast ID, an input time point, and a user ID, as shown in FIG. All are searched from the comment table (step S72). As a result, the extraction unit 150 allows the comment input when the program represented by the media information in the designated path is broadcast live or rebroadcast, the user who issued the comment, and the elapsed time from the broadcast start date and time. And the input time point of the comment represented by.

  Thereafter, the extraction unit 150 acquires a sentence (that is, an input sentence) that constitutes a comment for all of the searched comments (hereinafter referred to as a search comment), and the acquired input sentence is designated with the specified multimedia information. The sentence is related to the program being represented. Next, the extraction unit 150 generates a sentence set having the input sentence as an element (step S73).

  Thereafter, the extraction unit 150 searches the broadcast table shown in FIG. 5 for the shift time associated with the broadcast ID for each searched broadcast ID. Next, the extraction unit 150 generates a sentence ID of the input sentence. Thereafter, the searched shift time is set as the shift time corresponding to the input sentence of the comment searched with the same broadcast ID.

  Thereafter, the extraction unit 150 associates the generated sentence ID, the sentence, the kind of the sentence, the input time of the comment configured with the sentence, and the shift time corresponding to the sentence, 11 is stored in the sentence set table shown in FIG. 11 (step S74).

  The reason why the shift time is associated with the input sentence extracted from the comment is that it is presumed that the input timing of the comment differs from the output timing of the voice depending on the shift time. For this reason, it is necessary to associate the input sentence with the shift time for later processing.

  Thereafter, the extraction unit 150 searches the broadcast table shown in FIG. 5 for the broadcast start date and time associated with the broadcast ID for each broadcast ID searched in step S71 (step S75).

  Thereafter, the extraction unit 150 identifies the date and time when the comment was input (hereinafter referred to as the comment input date and time) by adding the searched broadcast start date and time to the input time for each of the comments searched at step S72 ( Step S76).

  Next, the extraction unit 150 calculates a time interval (hereinafter referred to as a comment input time) from a date and time that is a predetermined time A before a comment input date and time to a date and time that is a predetermined time B after the comment input date and time. Next, for each comment searched in step S72, the extraction unit 150 shows the URL associated with the reference date and time included in the comment input time and the user ID searched in step S72 in FIG. Search from the reference table (step S77). Accordingly, the extraction unit 150 identifies the document referred to by the user at the comment input time, and sets the identified document as the page referred to by the user for inputting the comment. Note that a suitable predetermined time A and predetermined time B can be determined by experiments by those skilled in the art.

  Next, the extraction unit 150 acquires the document at the URL for all the URLs searched in step S76 (step S78).

  Thereafter, the extraction unit 150 acquires a sentence (hereinafter referred to as a reference sentence) posted in the referenced document for all the acquired documents, and the acquired reference sentence is represented by designated multimedia information. The sentence is related to the program. Next, the extraction unit 150 adds the reference sentence to the sentence set (step S79).

  For example, the document that the viewer referred to while watching the program related to the program, such as the content that the viewer was wondering about or the content that the viewer wanted to check in the content broadcast on the program. This is because the contents are often posted.

  After that, the extraction unit 150 stores the reference sentence in the sentence set table shown in FIG. 11 (step S78), and ends the execution of the sentence set generation process. Specifically, the extraction unit 150 generates a sentence ID of a reference sentence, and inputs the generated sentence ID, the sentence, the type of the sentence, and a comment used for searching for a document including the sentence. And the shift time corresponding to the sentence are stored in the sentence set table in association with each other.

  The reason why the shift time is associated with the reference sentence extracted from the referenced document is that it is estimated that the reference timing of the document with respect to the output timing of the sound differs depending on the shift time. For this reason, it is necessary to associate the reference sentence with the shift time for later processing.

  After step S62 in FIG. 9, the extraction unit 150 extracts word candidates (that is, candidate words) representing the voice broadcast in the program from the sentences included in the sentence set, as shown in FIG. 16. Is executed (step S63).

  When the execution of the candidate word extraction process is started, the extraction unit 150 acquires all the sentences included in the sentence set (step S81). Next, the extraction unit 150 performs morphological analysis on each acquired sentence (step S82). Thereby, the extraction unit 150 extracts all of the words (that is, input words) constituting the input sentence and all of the words (that is, reference words) constituting the reference sentence from the respective sentences (step S83).

  Thereafter, the extraction unit 150 searches the co-occurrence word table shown in FIG. 12 for the co-occurrence words associated with the input words (that is, the input co-occurrence words) for each of the extracted input words. Next, the extraction unit 150 uses the input co-occurrence word searched based on the input word as the comment content of the performer of the program when the input word is input as part of the comment of the program. It is assumed that the word is assumed to be present (that is, co-occurs as the content of the statement).

  In addition, the extraction unit 150 searches the co-occurrence word table for the co-occurrence word (that is, the reference co-occurrence word) associated with the reference word for each of the extracted reference words (step S84). Next, when the viewer refers to the reference co-occurrence word searched based on the reference word in order to comment on the program, the extraction unit 150 is used for the remark of the performer of the program. It is assumed that the word is

  Next, the extraction unit 150 sets the input word and reference word extracted in step S83 and the input co-occurrence word and reference co-occurrence word searched in step S84 as candidate words (step S85).

  Thereafter, the extraction unit 150 stores the candidate words in the candidate word table shown in FIG. 13 (step S86), and ends the execution of the candidate word extraction process.

  Specifically, the extraction unit 150 generates a candidate word ID for identifying a candidate word for each candidate word. Next, the extraction unit 150 corresponds to an input word, a co-occurrence word of the input word, a reference word posted in a document searched based on a comment including the input word, and a co-occurrence word of the reference word This input time is set as the input time of the input sentence from which the input word is extracted.

  Next, the extraction unit 150 extracts the candidate word ID of the candidate word that is the input word, the candidate word, the type of the candidate word, the input time point corresponding to the candidate word, and the input sentence including the candidate word. The associated shift times are associated and stored in the candidate word table. The extraction unit 150 also inputs the candidate word ID of the candidate word that is the input co-occurrence word, the candidate word, the type of the candidate word, the input time point corresponding to the candidate word, and the co-occurrence. The shift time corresponding to the word is associated and stored in the candidate word table. Furthermore, the extraction unit 150 corresponds to a candidate word ID of a candidate word that is a reference word, the candidate word, a type of the candidate word, an input time point corresponding to the candidate word, and a reference sentence including the candidate word. The shifted shift time is stored in the candidate word table in association with each other. Furthermore, the extraction unit 150 estimates the co-occurrence of the candidate word ID of the candidate word that is the reference co-occurrence word, the candidate word, the type of the candidate word, the input time point corresponding to the candidate word. The shift times associated with the reference words are stored in the candidate word table in association with each other.

  After the candidate words are extracted in step S63 shown in FIG. 9, the speech recognition unit 160 shown in FIG. 4 calculates the likelihood of appearance for each candidate word (step S64).

Here, an example of the process of step S64 will be described.
The voice recognition unit 160 searches all candidate words stored in the candidate word table shown in FIG. Next, the speech recognition unit 160 assigns a first predetermined value as the appearance likelihood for each candidate word that is an input word. This first predetermined value appears in the audio of the program, for example, the input word is spoken in the program under the condition that the input word is input as part of the program comment. It is a value representing how much the likelihood of a thing is.

  In addition, the speech recognition unit 160 assigns a second predetermined value as an appearance likelihood for each candidate word that is a reference word. This second predetermined value is the degree of likelihood that the reference word appears in the audio of the program under the condition that the comment used for searching the reference word is input as part of the program comment. It is a value indicating whether or not. The first predetermined value and the second predetermined value can be set to suitable values by those skilled in the art through experiments.

  In addition, the extraction unit 150 generates a co-occurrence associated with the input word and the co-occurrence word from the co-occurrence word table illustrated in FIG. 12 for each of the co-occurrence words of the input word among the candidate words. Search for likelihood. Next, the extraction unit 150 assigns a value obtained by adjusting the first predetermined value using the searched co-occurrence likelihood (hereinafter, first adjustment value) as the appearance likelihood of the co-occurrence word. This first adjustment value is a value that represents the likelihood that the co-occurrence word will appear during the utterance of the program under the condition that a comment including the input word is input, The higher the co-occurrence likelihood, the higher the value.

  Furthermore, the extraction unit 150 generates a co-occurrence associated with the reference word and the co-occurrence word from the co-occurrence word table illustrated in FIG. 12 for each of the co-occurrence words of the reference word among the candidate words. Search for likelihood. Next, the extraction unit 150 assigns a value obtained by adjusting the second predetermined value using the searched co-occurrence likelihood (hereinafter, the second adjustment value) as the appearance likelihood of the co-occurrence word. This second adjustment value represents the degree of likelihood that the co-occurrence word will appear during the utterance of the program under the condition that the comment used to search for the reference word is input. This value is adjusted to a higher value as the co-occurrence likelihood is higher.

  After step S64 shown in FIG. 9 is executed, the input unit 120 reads out multimedia information of a predetermined size from the designated path specified in step S61 (step S65).

  Next, the voice recognition unit 160 shown in FIG. 4 recognizes the voice (hereinafter referred to as program voice) X of the program represented by the multimedia information read out in step S65, as shown in FIG. Recognition processing is executed (step S66).

  Note that the continuous speech recognition processing executed by the speech recognition unit 160 is described in Non-Patent Document 1, and will be briefly described below.

In the continuous speech recognition process, the probability p (W | X) that the content of the program audio X is represented by the word string W when the audio (hereinafter referred to as program audio) X of the program read in step S65 is input. ) To maximize the word string W ^* .

  Here, the probability p (W | X) can be rewritten as the following formula (1) by Bayes rule.

  Here, the denominator probability p (X) can be ignored because it is considered as a normalization coefficient that does not affect the determination of the word string W.

Therefore, the word string W ^* that maximizes the probability p (W | X) represented by the following formula (2) is also represented by the following formula (3) or formula (4).

In the present embodiment, the speech recognition unit 160 is described as searching for a word string W ^* that satisfies Expression (3), but is not limited thereto, and searches for a word string W ^* that satisfies Expression (4). You may do it.

  When the execution of the voice recognition process is started, the voice recognition unit 160 performs a program based on, for example, the frequency and the sound pressure from the voice signal of the voice represented by the multimedia information read in step S65 shown in FIG. Signal processing (hereinafter, referred to as program audio) X is performed (step S91).

Next, the voice recognizing unit 160 matches the frequency change of the extracted program audio X with the frequency pattern of phonemes and syllables represented by the acoustic model stored in the storage unit 190, so that the program The speech X is decomposed into phonemes and the like, and a phoneme sequence X = {x ₁ , x ₂ ,..., X _k } representing the program speech X is generated (step S92).

  Thereafter, the voice recognition unit 160 identifies the point of time when the program voice X was pronounced, and represents it using the elapsed time from the broadcast start date and time until the voice is emitted (step S93).

  Next, the speech recognizing unit 160, for all candidate words stored in the candidate word table shown in FIG. 13, the difference between the input time corresponding to the candidate word and the pronunciation time of the extracted program sound (that is, , Time difference) is calculated (step S94).

  Thereafter, the voice recognition unit 160 searches for the shift time corresponding to the candidate word for all of the candidate words stored in the candidate word table shown in FIG. Next, the voice recognition unit 160 is based on the time difference calculated in step S94 and the live broadcast match degree curve represented by the data stored in the storage unit 190 for candidate words whose shift time is equal to or less than a predetermined value. To calculate the degree of match. Also, the speech recognition unit 160 determines the matching degree based on the calculated time difference and the rebroadcast matching degree curve represented by the data stored in the storage unit 190 for the candidate words whose shift time is greater than the predetermined value. Is calculated (step S95).

  Next, the speech recognition unit 160 initializes the variable j used for counting the number of generated word strings W with the value “0” (step S96).

Next, the speech recognition unit 160 selects candidate words with higher probability as candidate words w ₁ to w _k constituting the word string W = {w ₁ , w ₂ ,..., W _k } with higher probability. . Also, the speech recognition unit 160 selects candidate words with higher probability of appearance as candidate words w ₁ to w _k constituting the word string W with higher probability. Then, the speech recognition unit 160 generates a word string W composed of candidate words _{w 1} selected in _{w k} (step S97). Note that the number k of candidate words constituting the word string W is stochastically determined when step S97 is executed.

Thereafter, the speech recognition unit 160 generates a phoneme sequence for each candidate word constituting the word sequence W using the word dictionary stored in the storage unit 190, and the phoneme sequence M = {representing the pronunciation of the word sequence W m ₁ , m ₂ ,..., m _i } are generated (step S98).

  Next, the voice recognition unit 160 calculates the probability p (X | W) that the program voice X occurs from the word string W using the following equation (5) (step S99). The probability p (X | W) is referred to as a degree of coincidence because it represents how much the phoneme sequence representing the pronunciation of the word sequence X matches the phoneme sequence of the program audio.

The voice recognition unit 160, compares the acoustic features of phonemes such as m _i represented by the acoustic model, the acoustic features of phonemes such as x _i decomposed from the audio signal, but to what extent coincide and, more consistent p _| a (x i _{m i)} is a value close to the value "1", p as are different _| a (x i _{m i)} to a value close to "0".

  Next, the voice recognizing unit 160 is the probability that the word string W will occur at the time when the program voice X is input, and the degree of connectivity p (W) representing the linguistic accuracy unrelated to the program voice X is calculated as follows: It calculates using the following formula | equation (6). At this time, the speech recognition unit 160 approximates the following equation (6) by equation (7), and calculates an approximate value of the coupling degree p (W) using the language model of the N-gram model (step S100). This is to reduce the calculation amount.

  After that, the voice recognition unit 160 calculates p (W | X) by multiplying p (X | W) calculated in step S99 and the degree of coupling p (W) calculated in step S100. (Step S101).

  Thereafter, the voice recognition unit 160 increases the variable j by the value “1” (step S102), and then determines whether or not the variable j is larger than the predetermined value Th (step S103). At this time, when the voice recognition unit 160 determines that the variable j is equal to or smaller than the predetermined value Th (step S103; No), the speech recognition unit 160 returns to step S97 and repeats the above processing. A suitable predetermined value Th can be determined by a person skilled in the art through experiments.

On the other hand, when the voice recognition unit 160 determines that the variable j is larger than the predetermined value Th (step S103; Yes), the maximum p (W | X) is calculated among the calculated Th word strings W. After specifying the word string W ^* that satisfies (that satisfies the expressions (2) to (4)) (step S104), the continuous speech recognition process is terminated.

After the continuous speech recognition process in step S66 shown in FIG. 9 is executed, the speech recognition processing unit 160 adds the recognized word string W ^* to the summary (step S67).

  Thereafter, the input unit 120 shifts the reading position of the electronic file in the above-described path backward by the size of the read multimedia information. Next, the input unit 120 determines whether or not the reading position is the last EOF of the electronic file (step S68). At this time, if the input unit 120 determines that the reading position is not EOF (step S68; No), the above processing is repeated from step S65.

  In step S68, if the input unit 120 determines that the reading position is EOF (step S68; Yes), the output unit 140 outputs the summary to the video card 107 shown in FIG. 2 (step S69). Thereafter, the video card 107 causes the LCD 108 to display the summary.

  Next, the output unit 140 stores the designated path and the text representing the audio summary represented by the multimedia information in the designated path in association with each other in the storage unit 190 (step S70), and then summarizes the summary. The generation process is terminated. This is because multimedia information can be searched based on keywords.

  Here, the comment on the sound output by the reproduction of the multimedia information often includes a word representing the content of the sound or a co-occurrence word of the word. For this reason, according to these configurations, the speech recognition apparatus 100 uses a word representing the content of speech as a word (that is, an input word) constituting a comment and a co-occurrence word of the word (that is, an input co-occurrence word). Therefore, it is possible to recognize speech more appropriately than in the past. That is, the speech recognition apparatus 100 can recognize the speech included in the multimedia information more appropriately than before by using the comments attached to the multimedia information.

  Further, a user who inputs a comment about the audio of a program often checks or confirms the meaning content of the statement in a document. For this reason, a document viewed by a user who has listened to multimedia information and input a comment often includes a word representing the content of audio output by reproducing the multimedia information or a co-occurrence word of the word. Therefore, according to these configurations, the speech recognition apparatus 100 converts the word (that is, the reference word) constituting the document referred to by the user and the co-occurrence word (that is, the reference co-occurrence word) of the word into the content of the voice. As a candidate for a word representing (ie, a candidate word), the speech can be recognized more appropriately than before.

  Furthermore, according to these configurations, since the speech recognition is performed based not only on the degree of coincidence between the phoneme recognized from the speech and the phoneme representing the pronunciation of the candidate word, but also on the appearance likelihood of the candidate word, The speech can be recognized with higher accuracy than the conventional speech recognition device that recognizes the speech based only on the above.

  Here, the sound generation time and the comment input time with respect to the sound often coincide with each other, for example, because they are usually not different by a predetermined time or more. For this reason, the speech recognition apparatus 100 recognizes speech based on the degree of coincidence between the input time corresponding to the candidate word and the pronunciation time when the speech is emitted, and the comment including the candidate word. Speech can be recognized with higher accuracy than before.

  Here, as mentioned above, viewers who have already watched a program on live broadcasts and viewers who have repeatedly watched the same program on rebroadcasts will be subject to comment rather than viewers who have watched the program for the first time on live broadcasts. There is a tendency to input comments at a time close to the time of sound generation. As shown in FIG. 14, the rebroadcast match level curve stored in the speech recognition apparatus 100 is located above the live broadcast match level curve in the time difference range from “−TD1” to “TD2”. Therefore, if the same candidate word and the time difference is the same value included in the range from "-TD1" to "TD2", the word entered or referenced in the rebroadcast or the co-occurrence word of the word is The probability of being adopted in the word string W generated by the continuous speech recognition process shown in FIG. 17 is higher than that of a word input in live broadcasting or a co-occurrence word of the word.

  In addition, as described above, viewers who have already watched a program on a live broadcast often input a comment at the time of re-broadcasting, near the point of time when the voice to be commented is pronounced. Further, as shown in FIG. 14, the rebroadcast match curve stored in the speech recognition apparatus 100 has a peak when the time difference is “0”, and attenuates as the time difference is away from “0”. For this reason, if the words are the same candidate words that are both input by re-broadcasting or co-occurrence words of the words, the one with the smaller difference between the pronunciation time and the input time is generated by the continuous speech recognition process. The probability of being adopted for the word string W to be processed is high.

  On the other hand, viewers of live broadcasts often input comments on the audio after listening to the audio of the performer. As shown in FIG. 14, the live broadcast match degree curve stored in the speech recognition apparatus 100 has a peak when the time difference is “TP”, and attenuates as the time difference becomes farther from “TP”. For this reason, if the same candidate word is a word that is input live in live broadcasts or a co-occurrence word of the word, the difference between the time of pronunciation and the time of input is closer to “TP”. There is a high probability of being adopted for the word string W generated by the processing. Thereby, the speech recognition apparatus 100 can perform speech recognition with higher accuracy than in the past.

  In the present embodiment, the communication network 10 shown in FIG. 1 has been described as being the Internet. However, the communication network 10 is not limited thereto, and may be a LAN (Local Area Network) or a public line network.

  In the present embodiment, the multimedia information is described as representing the video and sound of the program, but is not limited to this, and may represent only the sound of the program.

<Example 2>
Similar to the speech recognition apparatus 100 according to the first embodiment, the speech recognition apparatus 200 according to the second embodiment of the present invention configures the speech recognition system 1 illustrated in FIG. In the following, since differences from the first embodiment will be mainly described, descriptions of common points with the first embodiment will be omitted.

  Since the hardware configuration of the speech recognition apparatus 200 is the same as the hardware configuration of the speech recognition apparatus 200 according to the first embodiment, the description thereof is omitted.

Next, functions of the speech recognition apparatus 200 will be described.
The CPU of the speech recognition apparatus 200 according to the second embodiment executes the summary generation process illustrated in FIG. 18, so that the input unit 220, the storage unit 230, the output unit 240, the extraction unit 250, and the speech recognition illustrated in FIG. Unit 260 and co-occurrence likelihood calculating unit 270. In addition, the CPU of the speech recognition apparatus 200 functions as the storage unit 290 in cooperation with the hard disk 104. The input unit 220, the storage unit 230, the output unit 240, the extraction unit 250, the voice recognition unit 260, and the storage unit 290 are the input unit 120, the storage unit 130, the output unit 140, the extraction unit 150, the voice described in the first embodiment. The recognizing unit 160 and the accumulating unit 190 have the same functions.

  The co-occurrence likelihood calculating unit 270, for each user who uses the terminal devices 20 to 40, a word posted in a document referred to by the user, a co-occurrence word used together with the word in the document, and the co-occurrence The word co-occurrence likelihood is calculated.

  The accumulating unit 190 stores the co-occurrence word table shown in FIG. 20 instead of the co-occurrence word table shown in FIG. In this co-occurrence word table, a user ID, a word posted in a document referred to by the user identified by the user ID, a co-occurrence word of the word, the word and the co-occurrence word are commented or A plurality of pieces of data in which likelihoods that indicate how likely it is to be used together in a document (that is, co-occurrence) (hereinafter referred to as co-occurrence likelihoods) are stored.

  Next, the operation of the CPU performed by each function unit shown in FIG. 19 will be described.

  When the CPU of the speech recognition apparatus 200 inputs a signal corresponding to the summary generation instruction operation from the keyboard, it starts executing the summary generation process shown in FIG.

  When the execution of the summary generation process is started, the co-occurrence likelihood calculating unit 270 executes the co-occurrence likelihood calculating process for calculating the co-occurrence likelihood (step S60).

  In the co-occurrence likelihood calculation process, the co-occurrence likelihood calculation unit 270 searches for a URL associated with the user ID for each user ID stored in the reference table illustrated in FIG. Next, the co-occurrence likelihood calculating unit 270 acquires documents in the URL for all the searched URLs. Thereafter, the co-occurrence likelihood calculating unit 270, for all of the acquired documents, the posted word posted in the document, the co-occurrence word used together with the posted word and the document, and the co-occurrence word being the posted Calculate the number of co-occurrence used with the word. Thereafter, the co-occurrence likelihood calculating unit 270 calculates the co-occurrence likelihood for all combinations of the posted word and the co-occurrence word based on the number of co-occurrence. Next, the co-occurrence likelihood calculating unit 270 associates the user ID, the posted word, the co-occurrence word, and the co-occurrence likelihood with respect to the co-occurrence likelihood equal to or greater than a predetermined value, as illustrated in FIG. Save to co-occurrence table.

  After the process of step S60 shown in FIG. 18 is executed, the processes of step S61 to step S63 are executed.

  Thereafter, the speech recognition unit 260 calculates the appearance likelihood for each candidate word (step S64). At this time, when the candidate word is an input co-occurrence word, the speech recognition unit 260 specifies the user ID of the user who has input the input word co-occurring with the input co-occurrence word, and specifies the specified user ID and the input word The co-occurrence likelihood associated with the input co-occurrence word in the co-occurrence table shown in FIG. 20 is searched. Next, the speech recognition unit 260 calculates an appearance likelihood using the searched co-occurrence likelihood. In addition, when the candidate word is a reference co-occurrence word, the speech recognition unit 260 identifies the user ID of the user who referred to the reference word that co-occurs with the reference co-occurrence word, and identifies the identified user ID, the reference word, The co-occurrence likelihood associated with the reference co-occurrence word in the co-occurrence table shown in FIG. 20 is searched. Next, the speech recognition unit 260 calculates an appearance likelihood using the searched co-occurrence likelihood.

  Thereafter, the voice recognition unit 260 ends the execution of the summary generation process after executing the processes from step S65 to step S70.

  According to these configurations, the speech recognition unit 200 uses the posted word posted in the document referred to by the viewer and the word used in the document together with the posted word as a co-occurrence word, and the posted word and the co-occurrence The co-occurrence likelihood is calculated based on the number of times the word co-occurs in the document. Further, the speech recognition 200 uses the calculated co-occurrence likelihood to calculate the appearance likelihood of the co-occurrence word of the word input or referred to by the viewer, and the calculated co-occurrence word appearance likelihood and the co-occurrence The speech is recognized based on the degree of coincidence between the pronunciation of the word and the speech. Here, the words that viewers co-occur with each other and use in comments and the words that co-occur with each other and appear in the document vary depending on the topic, trends in the times, and viewer preferences. For this reason, the speech recognition apparatus 200 can recognize speech with high accuracy even if the topic, trend of the times, or viewer's preference changes.

<Example 3>
The speech recognition apparatus 100 according to the first embodiment generates a video in which a comment is synthesized in step S17 shown in FIG. 3, and multimedia information representing the video in which a comment is synthesized in step S19 is shown in FIG. It has been described that the data is output to the LAN card 106. The LAN card 106 transmits the multimedia information to the terminal devices 20 and 30, and the terminal devices 20 and 30 explain that the combined video is displayed in the video display area AM of the viewing screen shown in FIG. did.

  The speech recognition apparatus according to the third embodiment does not generate a moving image in which the comment is synthesized in step S <b> 17 illustrated in FIG. 3, and outputs the multimedia information and the comment information to the LAN card 106 in step S <b> 19. The LAN card 106 transmits the multimedia information and the comment information to the terminal device.

  The terminal device according to the third embodiment displays a viewing screen as illustrated in FIG. This viewing screen is superimposed on the moving image display area AM, the comment display area AC, and the moving image display area AM described in the first embodiment (that is, the comment display belongs to a higher layer than the moving image display area AM). And a column UL. When the terminal device receives the multimedia information and the comment information, the terminal device displays the moving image represented by the multimedia information in the moving image display area AM, and displays the comment represented by the comment information superimposed on the moving image display area AM. Displayed in the column UL and the comment display area AC. The frame of the comment display field UL is indicated by a dotted line for the convenience of drawing, but the frame of the comment display field UL is not displayed on the viewing screen.

<Example 4>
The speech recognition apparatus 100 according to the fourth embodiment not only broadcasts and re-broadcasts a program, but also distributes the program by VOD (Video On Demand). The terminal devices 20 to 40 display not only a live broadcast or rebroadcast program but also a video of the distributed program and output sound.

  Here, a description will be given on the assumption that the user of the terminal device 40 performs an operation for causing the terminal device 40 to transmit a request (hereinafter referred to as a VOD distribution request) requesting distribution of a live broadcast program by VOD.

  The terminal device 40 transmits a VOD distribution request to the voice recognition device 100 according to the operation. When receiving the VOD distribution request from the terminal device 40, the voice recognition device 100 reads out multimedia information representing the program requested to be distributed, and starts distributing the read multimedia information to the terminal device 40. The terminal device 40 starts displaying the program video represented by the multimedia information received from the voice recognition device 100 and outputting the program audio.

  Then, it demonstrates that the user of the terminal device 40 performs the skip operation which advances the reproduction | regeneration position of the delivered program ahead predetermined time to the terminal device 40. FIG.

  The terminal device 40 stops the display of the program video and the output of the program audio, and transmits a skip command for instructing the skip and the time to skip to the voice recognition device 100. When the voice recognition device 100 receives the skip command, it shifts the read position of the multimedia information backward by the size corresponding to the time specified by the skip command, and then continues to read and distribute the multimedia information. Thereafter, the terminal device 40 resumes the display of the program video and the output of the program audio represented by the distributed multimedia information.

  After that, when the user of the terminal device 40 performs a skip operation on the terminal device 40 to return the playback position of the distributed program after a predetermined time, the terminal device 40 stops displaying the program video and outputting the program audio, Using the saved multimedia information, the reproduction of the program video and the output of the program audio are resumed from the previous reproduction position by the size corresponding to the time designated by the skip operation.

  When the user of the terminal device 40 performs an operation on the terminal device 40 to temporarily stop the reproduction of the distributed program, the terminal device 40 stops displaying the program video and outputting the program audio. After that, when the user of the terminal device 40 performs an operation for frame-by-frame playback of the distributed program, the terminal device 40 stops outputting the program audio, and uses the distributed or already stored multimedia information. Start frame-by-frame playback of program video.

  Thereafter, when the user of the terminal device 40 performs an operation on the terminal device 40 to stop the reproduction of the program, the terminal device 40 stops the display of the program video and the audio output, and issues a stop command instructing the stop to the voice recognition device. To 100. When receiving the stop command from the terminal device 40, the voice recognition device 100 stops the delivery of the multimedia information according to the stop command.

  Examples 1 to 4 can be combined with each other. In addition to being able to be provided as a speech recognition apparatus 100 having a configuration for realizing the function according to any one of the first to fourth embodiments, the system includes a plurality of apparatuses, and any one of the first to fourth embodiments. It is also possible to provide a system having a configuration for realizing the functions according to the system as a whole.

  Note that the speech recognition apparatus 100 provided with a configuration for realizing the function according to the first embodiment in advance, the speech recognition apparatus 200 provided with a configuration for realizing the function according to the second embodiment in advance, or the third or fourth embodiment. In addition to being able to be provided as a speech recognition device having a configuration for realizing the functions related to the above, the existing speech recognition device can be replaced with the speech recognition device 100 according to the first embodiment and the speech recognition according to the second embodiment by applying a program. It can also be made to function as the apparatus 200 or the voice recognition apparatus according to the third or fourth embodiment. That is, a speech recognition program for realizing each functional configuration by the speech recognition device 100 exemplified in the first embodiment, the speech recognition device 200 exemplified in the second embodiment, or the voice recognition device exemplified in the third or fourth embodiment, By being applied so that a computer (such as a CPU) that controls an existing speech recognition device can be executed, the speech recognition device 100 according to the first embodiment, the speech recognition device 200 according to the second embodiment, or the third or fourth embodiment. The voice recognition apparatus can function.

  Such a program distribution method is arbitrary. For example, the program can be distributed by being stored in a recording medium such as a memory card, a CD-ROM, or a DVD-ROM, or via a communication medium such as the Internet. . The speech recognition method according to the present invention can be implemented using the speech recognition apparatus 100 according to the first embodiment, the speech recognition apparatus 200 according to the second embodiment, or the speech recognition apparatus according to the third or fourth embodiment.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

10: communication network 20, 30, 40: terminal device 21: photographing device 22: voice collection device 100, 200: voice recognition device 101: CPU
102: ROM
103: RAM
104: Hard disk 105: Media controller 106: LAN card 107: Video card 108: LCD
109: keyboard 110: speaker 111: touch pad 120, 220: input unit 130, 230: storage unit 140, 240: output unit 150, 250: extraction unit 160, 260: voice recognition unit 190, 290: storage unit 270: shared Likelihood calculation unit

Claims (8)

An accumulator for accumulating comments input by the user while listening to audio generated by playing multimedia information;
An extraction unit that extracts words that appear in a sentence set including the accumulated comments and candidate words including co-occurrence words of the word in the sentence set;
A voice recognition unit that recognizes a voice generated by reproducing the multimedia information based on the extracted candidate word;
A speech recognition apparatus characterized by that. The speech recognition device according to claim 1,
The speech recognition apparatus, wherein the sentence set includes a sentence that appears in a document viewed by a user who has listened to the multimedia information. The speech recognition apparatus according to claim 1 or 2,
The extraction unit calculates the likelihood of appearance of each of the candidate words,
The speech recognition unit recognizes speech based on the degree of coincidence between the phoneme recognized from the speech and the phoneme representing the candidate word and the appearance likelihood of the candidate word.
A speech recognition apparatus characterized by that. The speech recognition device according to claim 3,
Of the candidate words, words appearing in the comment are associated with an input time point when the comment is input,
For the candidate word associated with the input time point, the speech recognition unit obtains a degree of match between the input time point associated with the candidate word and the pronunciation time point when the phoneme is emitted, Voice recognition based on the degree of match
A speech recognition apparatus characterized by that. The speech recognition device according to claim 4,
The input time point and the sound generation time point are expressed by a reproduction time after the reproduction of the multimedia information is started.
A speech recognition apparatus characterized by that. The speech recognition device according to claim 5,
The degree of match is determined based on a difference between the input time point and the sound generation time point, and a difference between a time point when the multimedia information can be played back and a time point when the user starts playing the multimedia information.
A speech recognition apparatus characterized by that. Computer
An accumulator for accumulating comments input by the user while listening to audio generated by playing multimedia information;
An extraction unit that extracts words that appear in a sentence set including the accumulated comments and candidate words including co-occurrence words of the word in the sentence set;
Based on the extracted candidate words, function as a speech recognition unit that recognizes speech generated by playing the multimedia information,
A speech recognition program characterized by that. A method performed by a speech recognition apparatus including an accumulation unit, an extraction unit, and a speech recognition unit,
An accumulating step in which the accumulating unit accumulates a comment input by the user while listening to a sound uttered by reproduction of multimedia information;
An extraction step in which the extraction unit extracts a word that appears in a sentence set including the accumulated comments and a candidate word including a co-occurrence word of the word in the sentence set;
The speech recognition unit has a speech recognition step for recognizing speech generated by reproducing the multimedia information based on the extracted candidate words.
A speech recognition method characterized by the above.

Images