JP6322125B2 - Speech recognition apparatus, speech recognition method, and speech recognition program - Google Patents

Description

  The present invention relates to a technique for performing speech recognition in consideration of background speech.

  Needless to say, providing high-accuracy input means by voice in a service on the web is important from the viewpoint of convenience for the user. In recent years, as the terminals used for browsing the web by users have diversified, the processing capabilities thereof vary, and some terminals are not suitable for performing advanced calculation processing. For this reason, the speech recognition process is often performed by a server in the server / client model. Research has been conducted on methods for enhancing the tolerance to noise and improving accuracy when performing speech recognition in the server / client model.

  Further, in recent years, a specific processing method of speech recognition that has become mainstream is based on statistical machine learning, and outputs a probabilistic likelihood as a result of recognition. That is, the recognition result is only an estimate (see Non-Patent Document 1).

Tatsuya Kawahara, “Study on Methodology of Speech Recognition-Toward a Change of Generations”, Information Processing Society of Japan (SLP), Spoken Language Information Processing, 2014-SLP-100 (3), pp.1-5, 2014.

  There are many Japanese pronunciations that have the same sound but different characters. For example, “Sendai” has the place names “Sendai (Miyagi Prefecture)” and “Kawauchi (Kagoshima Prefecture)”. Cannot be distinguished, and estimation is difficult when the input information in speech recognition is only speech data. Also in English, there are some that are spelled in the same sound but different in spelling.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for improving the accuracy of speech recognition and presenting a speech recognition result intended by a user. .

In order to solve the above-described problems, the present invention is a speech recognition device, in which speech data is identified by using a phase difference information to determine a distance to a sound source, and speech audio data uttered by a user and background The audio feature information is extracted from the background audio data separated into the audio data, and the content used by the user using the audio feature storage unit in which the audio feature information of a plurality of contents is stored. a specifying unit for specifying, by recognizing speech the speech data, text data is converted into at least one text candidate data, using said relevant information about the identified content, is presented to the user from among the text candidate data A voice recognition unit for determining

The present invention is a speech recognition method performed by a speech recognition apparatus, wherein speech data is classified into speech speech data uttered by a user and background speech data by specifying a distance to a sound source using phase difference information. Separating step for separating and extracting audio feature information from the separated background audio data, and specifying the content used by the user using the audio feature storage unit storing the audio feature information of multiple contents a specifying step of, by recognizing speech the speech data, a conversion step of converting at least one text candidate data, using the related information about the content that the identified and presented to the user from among the text candidate data And a determination step for determining text data.

  The present invention is a speech recognition program that causes a computer to function as the speech recognition apparatus.

  According to the present invention, it is possible to improve the accuracy of speech recognition and provide a technique for presenting a speech recognition result intended by a user.

It is a block diagram which shows the structure of the speech recognition system of embodiment of this invention. It is a sequence diagram which shows the process of a speech recognition system. It is a flowchart which shows a content specific process. It is a figure which shows an example of related information DB. It is a figure which shows the specific example of a speech recognition process. It is a block diagram which shows the structure of the information delivery system which is a modification of this embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  While viewing content (for example, a broadcast program such as a television) while searching for information related to the content on the web, it is generally performed now. Here, if voice input is used for the search, the input voice data is the voice (utterance voice) that is a search request of the user (user), and the voice of the content flowing behind is also used as the background voice. Included in the data. In the present embodiment, the background voice is used to identify the content that the user is currently using, and the information related to the content can be used to recognize the details of the search request input by voice with high accuracy. Information source.

  FIG. 1 is a configuration diagram showing the configuration of the speech recognition system of the present embodiment. The illustrated voice recognition system includes a client 1 and a server 2 (voice recognition device). In the present embodiment, the client 1 assumes the use of the voice recognition function on the server 2 side (on the web page) without providing the voice recognition function.

  Currently, browsing of web pages is generally performed through a browser, so the client 1 is a browser. The web page is usually configured by HTML, CSS, and JavaScript, and each function unit described below is realized in the client 1 by a program such as JavaScript.

  The client 1 is a user terminal used by a user, and a smartphone, a tablet terminal, a PC, or the like can be used. The illustrated client 1 includes an audio acquisition unit 11, a position acquisition unit 12, a communication unit 13, and a result display unit 14.

  The voice acquisition unit 11 acquires voice data using a microphone (not shown) included in the client 1. Conventionally, in browsers, it has been common to install special software called plug-ins or the like in advance in order to obtain sound from a microphone. By using API (Application Programming Interface) that can be used from JavaScript on the browser included in HTML5 in a broad sense, it is possible to obtain audio from a microphone without installing special software. It was.

  Here, the fact that it can be realized by utilizing the API that can be used from the above JavaScript means that it is possible to acquire sound from the microphone only by the description in the web page distributed from the web server. Specifically, by using getUserMedia and Web Audio API, the audio acquisition unit 11 can pass the acquired audio data to the communication unit 13 in a streaming format.

  The position acquisition unit 12 acquires position information (location information) of the client 1. As with the sound acquisition unit 11, the position acquisition unit 12 can also acquire position information by using an API that can be used from JavaScript. Specifically, the current location information can be acquired by using the Geolocation API and using the navigator.geolocation.getCurrentPosition function.

  The communication unit 13 transmits the audio data acquired by the audio acquisition unit 11 and the position information acquired by the position acquisition unit 12 to the server 2 via the network. Note that the communication unit 13 transmits the audio data to the server 2 in a streaming format. Here, WebSocket, which is one of HTML5 functions, is used for streaming transmission. As for the position information, the communication unit 13 transmits the position information to the server 2 as soon as the position acquisition unit 12 can acquire it, that is, at the timing when the position acquisition unit 12 acquires the position information. In addition, the communication unit 13 receives information transmitted from the server and sends it to the result display unit 14.

  The result display unit 14 displays various information such as recognition results received from the server 3 via the communication unit 13 on a display (not shown).

  In the present embodiment, the voice acquisition unit 11 is the voice data (background voice data) of the content (viewing content) flowing behind before the voice data (speech voice data) input by the user's utterance is input. ) And the communication unit 13 continues to transmit the audio data to the server 2. As a result, the amount of audio data for specifying the content that the user is viewing increases, and the accuracy of content specification can be improved.

  The server 2 performs voice recognition on the voice data transmitted from the client 1 and provides the recognition result to the client 1. Note that the server 2 has no restrictions on the implementation language. The illustrated server 2 includes a communication unit 21, a content specifying unit 22, a voice recognition unit 23, a voice feature DB (database) 24, and a related information DB 25.

  The communication unit 21 receives the audio data and position information transmitted from the client 1, sends the received audio data and position information to the content specifying unit 22, and transmits the audio data transmitted from the client 1 to the audio recognition unit 23. To send. In addition, the communication unit 21 transmits the recognition result of the voice recognition unit 23 to the client 1.

  The content specifying unit 22 extracts audio feature information from background audio data attached to the audio data input by the user, and specifies the content used by the user using the audio feature DB 24. In the present embodiment, the content specifying unit 22 acquires the location information of the user, narrows down broadcast programs according to the location information from the content stored in the audio feature DB 24, and features the narrowed content. The content that the user is viewing is specified by collating the information with the feature information of the background audio data. That is, after the accuracy is improved using the position information, the content used by the user is specified using the audio feature DB 24 based on the audio data. The content specifying unit 22 sends information for identifying the specified content (for example, a content ID) to the voice recognition unit 23.

  The speech recognition unit 23 recognizes speech data, converts the speech data into text candidate data that is at least one recognition result candidate, and uses text related to the identified content to present text to the user from the text candidate data. Determine the data. Specifically, the voice recognition unit 23 performs voice recognition of voice data uttered by the user received from the communication unit 21, acquires related information of the content specified by the content specifying unit 22 from the related information DB 25, and stores it in the content. Increase the probability of selecting relevant text candidate data. Then, the speech recognition unit 23 transmits the selected text candidate data to the client 1 via the communication unit 21.

  The audio feature DB 24 stores audio feature information of a plurality of contents. The related information DB 25 stores related information regarding a plurality of contents.

  Note that the client 1 and the server 2 described above can use a general-purpose computer system including a CPU, a memory, an external storage device such as a hard disk, an input device, and an output device, for example. In this computer system, each function of each unit is realized by the CPU executing a predetermined program loaded on the memory. For example, each function of the client 1 and the server 2 is realized by the CPU of the user terminal 1 being executed in the case of the program for the client 1 and the CPU of the server 2 being executed in the case of the program for the server 2. .

  The client 1 program and the server 2 program can be stored in a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, a DVD-ROM, or distributed via a network. it can.

  Below, the process of this embodiment is demonstrated.

  FIG. 2 is a sequence diagram showing processing of the voice recognition system. The voice acquisition unit 11 of the client 1 acquires voice data from the microphone (S1) and transmits it to the server 2 using the communication unit 13 (S2). Note that the audio data includes utterance audio data input by the user speaking and background audio data of the content used (viewed) by the user. Note that the audio data is transmitted to the server 2 in a streaming format.

  Further, the position acquisition unit 12 of the client 1 acquires position information (S3) and transmits it to the server 2 using the communication unit 13 (S4). The processing of S1 and S2 and the processing of S3 and S4 are performed asynchronously, and the processing of S3 and S4 is performed before the processing of S1 and S2, or the processing of S1 and S2 The order of processing is not limited to the example shown in FIG.

  The content specifying unit 22 of the server 2 extracts the audio feature information from the background audio data of the audio data transmitted from the client 1, and specifies the content used by the user using the audio feature DB 24 (S5). . Then, the speech recognition unit 23 of the server 2 recognizes speech data of the speech data transmitted from the client 1 and converts it into at least one recognition result candidate converted into text, and refers to the related information DB 25, The text data to be presented to the user is determined from the text candidate data using the related information regarding the identified content (S6). The processing of S5 and S6 will be described later.

  The voice recognition unit 23 transmits the determined text data to the client 1 using the communication unit 21 (S7). The result display unit 14 of the client 1 displays the recognition result received from the server 2 using the communication unit 13 on the display (S8).

  Below, the process of S5 and S6 is demonstrated in detail.

  In the content specifying process of S5, the content used by the user of the client 1 is specified using the audio data and the position information obtained via the communication unit 21. Here, as a method of specifying content using audio data, a method called audio fingerprint is known among automatic content recognition (ACR) techniques.

  The voice fingerprint is described in Japanese Patent Application Laid-Open No. 2004-326050 (hereinafter referred to as “Document 1”. Specifically, the voice data is set to a certain time length, for example, about 20 milliseconds to 40 milliseconds. After dividing, the feature information that is a characteristic index for the speech section is extracted as a multidimensional vector, and by comparing the extracted feature information with the feature information of each content stored in the speech feature database, In this case, for example, a mel frequency cepstrum coefficient can be used as the multidimensional vector.

  Here, the case where the content is a broadcast program such as a television will be described below as an example. In this case, the audio feature DB 24 stores the feature information of the audio data of the program currently being broadcast. The server 2 (or an external system (not shown)) receives each broadcast program by each broadcast wave, divides the audio data of each broadcast program at a certain time length as described above, and then enters the corresponding audio section. Feature information that is a characteristic index is extracted and stored in the voice feature DB 24 sequentially. Although a predetermined time lag occurs between the feature information obtained from the sound data transmitted from the client 1 and the feature information of the sound data of each broadcast program stored in the sound feature DB 24, the user at the client 1 The audio data of the broadcast program being viewed is transmitted to the server 2 in almost real time, the feature information of the audio data is extracted, and collated with the feature information of each program in the audio feature DB 24. Then, even if it is a broadcast program currently being broadcast, it is possible to specify the program that the user is viewing.

  The audio feature DB 24 stores a program ID and audio data feature information in association with each broadcast program.

  The audio feature DB 24 may store the audio data of each broadcast program, not the feature information of the audio data of each broadcast program. In this case, the content specifying unit 22 extracts the feature information of each broadcast program stored in the audio feature DB 24 in the content specifying process of S5, and collates it with the feature information of the audio data transmitted from the client 1. To do.

  FIG. 3 is a flowchart showing the flow of content identification processing (S5) that the user is viewing. Here, a case where the content is a broadcast program will be described as an example.

  First, the content specifying unit 22 determines whether or not the position information transmitted from the client 1 has been acquired (S11). If the position information has been acquired (S11: YES), the position belongs based on the position information. Narrow down (ie identify) broadcast programs currently being broadcast in the region. Then, the content specifying unit 22 extracts audio feature information from the audio data (background audio data) transmitted from the client 1, and extracts the feature information from the narrowed down broadcast program feature information in the audio feature DB 24. Each is collated to estimate the broadcast program that the user is currently viewing (S12).

  That is, the content specifying unit 22 compares each broadcast program narrowed down in the audio feature DB 24 with the feature information of the audio data, and calculates a probability value (also referred to as a certainty factor) as a comparison result. In the content specifying unit 22, it is assumed that only the background audio data is included in a portion having a low signal strength by using the signal strength information within a predetermined time of the audio data received from the client 1 through the communication unit 21. By using only that data, the accuracy of content identification can be improved. Furthermore, when the audio data received from the client 1 through the communication unit 21 is composed of a plurality of channels (stereo audio data), the sound source separation is performed by specifying the distance to the sound source using the information of the phase difference. The voice data component that is determined to be far away is used as the background voice to improve the accuracy of content identification, and the voice recognition is performed as the utterance part that should recognize the component that is close to the distance. It is also possible to increase the accuracy.

  When there is a broadcast program larger than a predetermined probability value determined in advance (S13: YES), the content identification unit 22 identifies the broadcast program as a viewing program that the user is currently viewing, and the viewing program Is output to the voice recognition unit 23 (S15).

  On the other hand, when there is no program larger than the predetermined probability value among the broadcast programs broadcast in the area (S13: NO), the content specifying unit 22 is characterized by all the broadcast programs stored in the audio feature DB 24. The information is collated with the feature information of the voice data transmitted from the client 1, and each probability value is calculated (S14). Then, the content identification unit 22 identifies the broadcast program with the highest probability value as the viewing program that the user is currently viewing, and outputs a program ID that identifies the viewing program to the voice recognition unit 23 (S15). ).

  Note that the user does not always watch the program that is currently being broadcast, but may be performing a so-called time-shifted viewing that plays back the recorded program. When the feature information of the past broadcast program is stored in the audio feature DB 24, it is possible to identify the past program by collating with the feature information of all the broadcast programs including the past stored in the audio feature DB 24. Also, it is possible to specify the viewing program of the user.

  Next, in the speech recognition process of S6, the speech recognition unit 23 recognizes speech speech data input by speech by the user from the speech data transmitted from the client 1, and communicates the recognition result to the communication unit 21. Return (send) to the client 1 via In this speech recognition process, generally, a plurality of recognition results (N-Best solutions) are obtained with probability values in the following two aspects.

  First, for speech data to be recognized, if the speech data is unclear due to noise or other noise when recognizing the utterance content as a sequence of hiragana characters in Japanese, it is accompanied by a probability value. There are multiple candidates. Second, even when an appropriate kanji is assigned to the recognized hiragana and used as the final recognition result, a plurality of candidates are generated due to the problem of homophones.

  In the present embodiment, the content specified by the content specifying unit 22 and stored in the related information DB 25 as an information source for drawing the user's intention out of a plurality of recognition result candidates with probability values. Use related information (words or specific expressions) related to the content.

  FIG. 4 is a diagram illustrating an example of the related information DB 25. In the related information DB 25 shown in the figure, for each broadcast program, a program ID, a program name, a broadcast date and time, a broadcast station name, and at least one related information are stored in association with each other in a table format. In the related information, for example, performer names, place names, and other keywords related to other programs are set. The program name, broadcast date and time, and broadcast station name are also part of the related information.

  When there are a plurality of candidates as a recognition result, the speech recognition unit 23 reads the related information of the broadcast program specified by the content specifying unit 22 from the related information DB 25 and uses the read related information from the plurality of candidates. The recognition result intended by the person is selected.

  FIG. 5 shows a specific example of processing when the speech recognition unit 23 selects a recognition result intended by the user from a plurality of candidates.

  In the example shown in the figure, the user inputs an operation instruction to the client 1 by speaking to the client 1 as “a popular pub in Sendai”. Speech voice data input to the client 1 is transmitted to the server 2 together with background voice data. Here, it is assumed that a user speaks when “Sendai” is pronounced and is input to the client 1 with an unclear pronunciation.

  The voice recognition unit 23 of the server 2 extracts utterance voice data from the voice data transmitted from the client 1 (S21), recognizes the utterance voice data and converts it into text, and recognizes a plurality of hiragana recognition result candidates shown in the figure. (Text candidate data) is acquired (S22). That is, the recognition result candidate 1 “Sendai Nichinki Izakaya” and the recognition result candidate 2 “Sendai Garlic Izakaya” are generated as recognition results.

  On the other hand, in the illustrated example, it is assumed that the program ID “200002” (Branched Miyagi Journey) shown in FIG. Thereby, the voice recognition unit 23 reads related information (for example, program name, performer name, place name, etc.) stored in association with the program ID from the related information DB 25 of FIG. 3 (S23).

  Then, the speech recognition unit 23 assigns appropriate kanji to the hiragana recognition result candidates 1 and 2 recognized in S22 (S24). In the illustrated example, three recognition result candidates (kanji characters) are generated. That is, the recognition result candidate 1 “Izakaya popular in Sendai”, the recognition result candidate 2 “Popular pub in Kawauchi”, and the recognition result candidate 3 “Popular pub in the ship” are generated.

  Then, the speech recognition unit 23 selects “Sendai” that matches (or is related to) any of the related information read from the related information DB 25 from among the plurality of candidate words “Sendai”, “Kawauchi”, and “Shipboard”. It is determined that the recognition result candidate 1 “Izakaya popular in Sendai” converted into “” is the recognition result intended by the user. Then, the voice recognition unit 23 transmits the determined “popular pub in Sendai” (text data) to the client 1 using the communication unit 21.

  In the above specific example, it is described that the speech recognition unit 23 selects a recognition result candidate that matches any of the related information read from the related information DB 25, but the recognition that is presented to the user using the probability value. The result can also be determined. For example, when the probability value of the recognition result candidate 1 is α, the probability value of the recognition result candidate 2 is β, and the probability value of the recognition result candidate 3 is γ, it matches any of the related information read from the related information DB 25 ( Alternatively, the probability value α of the recognition result candidate 1 (related) is adjusted to be large. For example, it is conceivable that the probability value of the recognition result candidate 1 is α × n or α + n using a predetermined coefficient n. The speech recognition unit 23 can also consider a method of presenting the recognition result candidates to the user after rearranging the plurality of recognition result candidates in descending order of probability values. The speech recognition unit 23 may determine a recognition result candidate having the largest probability value as a recognition result to be presented to the user from among a plurality of recognition result candidates.

<Modification>
In the embodiment described above, the method for improving the accuracy of voice recognition in consideration of the content used by the user has been described focusing on the case where the content is a broadcast program. A modification in which information related to a user is distributed by applying this voice recognition system is also conceivable.

  FIG. 6 is a diagram illustrating an example of a modified information distribution system.

  The information distribution system shown in FIG. 6 is a system to which the server 2 (voice recognition device) shown in FIG. 1 is applied. Specifically, although one server 2 is shown in FIG. 1, it is more efficient to divide the server into a unit of function rather than consolidating the functions into one server in actual system construction and operation. Sometimes. For this reason, in the modification shown in FIG. 6, it is assumed that the information distribution apparatus is configured by three servers 2A, 2B, and 2C.

  Here, the difference from the server 2 in FIG. 1 is the information presentation unit 26 and the information DB 27 of the server 2A. The information presenting unit 26 uses the content specified by the content specifying unit 22, the voice recognition result by the voice recognition unit 23, and the user's position information transmitted from the client 1 to provide useful information for the user. Present and distribute. The information DB 27 stores various types of information provided to the user.

  In other words, in the information distribution system according to the modified example, the user views and listens to the information DB in which various information is stored by general voice input and presents the information to the user. By identifying the content, it is possible to obtain a voice recognition result that eliminates the problem of homonyms and to present information that better captures the user's intention, as well as to use the user's location information By doing so, it is possible to edit the information to be presented or to adjust the priority order, and it is possible to further improve the accuracy and quality of the information to be presented.

  As an example, when the information distribution system shown in FIG. 6 is mainly for accommodation facility information such as hotels and inns, the information included in the information DB 27 is naturally accommodation facility information. As shown in the example of FIG. 5 described above, when the broadcast program being viewed is a travel program related to Miyagi Prefecture, and the voice recognition result is “Ryokan in Miyagi Prefecture”, the information presentation unit 26 determines the location of the user. Using the information, the result retrieved from the information DB 27 is edited. For example, when the location information of the user is Miyagi Prefecture, the information presentation unit 26 has an accommodation facility for which there is a day trip plan rather than presenting an accommodation plan for the accommodation facility in Miyagi Prefecture retrieved from the information DB 27. It is determined that it is more effective to preferentially present the information, and the priority of the information on the accommodation facility where the day plan exists is set high. On the other hand, when the location information of the user is Osaka, the information presenting unit 26 determines that it is more appropriate to present only the accommodation plan, and the day plan may be deleted from the search result. .

  In the present embodiment described above, the content used by the user is specified using background audio data attached to the audio data input by the user, and the user inputs it in consideration of related information related to the specified content. Speech data is recognized and converted to text data. Thereby, in this embodiment, the accuracy of speech recognition can be improved and the recognition result of speech data intended by the user can be provided.

  In the present embodiment, the position information of the client 1 is used when specifying the content being used from the background audio in the audio data. As a result, it is possible to increase the accuracy of content identification and increase the efficiency, and further improve the accuracy of speech recognition.

  In addition, this invention is not limited to the said embodiment, A various change and application are possible within a claim. For example, in the above-described embodiment, a mode in which a broadcast program is used as background audio has been described as an example. However, what can be used as background audio is not limited to audio data of a broadcast program. For example, when browsing a music information web page while listening to music, identify the music you are listening to, consider the location information of the user, and hold event information in the vicinity of the music player If there is, you can use it by presenting it.

DESCRIPTION OF SYMBOLS 1: Client 11: Voice acquisition part 12: Position acquisition part 13: Communication part 14: Result display part 2: Server 21: Communication part 22: Content identification part 23: Voice recognition part 24: Voice feature DB
25: Related information DB
26: Information construction department

Claims (5)

A speech recognition device,
By identifying the distance to the sound source using the phase difference information, the voice data is separated into the speech voice data uttered by the user and the background voice data, and the feature information of the voice is obtained from the separated background voice data. A specifying unit for extracting and specifying content used by a user using an audio feature storage unit in which audio feature information of a plurality of contents is stored;
The speech data by voice recognition, was converted into at least one text candidate data, using said relevant information about the identified content, the text candidate speech recognition unit for determining the text data to be presented to the user from the data And a voice recognition device comprising: The speech recognition apparatus according to claim 1,
The speech recognition apparatus characterized in that the content is a broadcast program. The speech recognition device according to claim 2,
The specifying unit acquires user position information, narrows down broadcast programs according to the position information from the contents stored in the audio feature storage unit, feature information of each narrowed broadcast program, and the background A speech recognition apparatus characterized by identifying broadcast programs viewed by a user by collating with feature information of speech data. A speech recognition method performed by a speech recognition device,
A separation step of separating the voice data into speech voice data uttered by the user and background voice data by specifying a distance to the sound source using phase difference information;
A step of extracting audio feature information from the separated background audio data, and specifying content used by the user using an audio feature storage unit in which audio feature information of a plurality of contents is stored;
The speech data by speech recognition and a conversion step of converting at least one text candidate data,
And a determination step of determining text data to be presented to a user from among the text candidate data using related information relating to the specified content.   A speech recognition program for causing a computer to function as the speech recognition apparatus according to any one of claims 1 to 3.

Images