JP2022048516A - Information processing unit, program and information processing method - Google Patents

Abstract

To provide an information processing unit, a program and an information processing method that can precisely record details of a speech of a speaker even if a speaker's voice is hard to recognize.SOLUTION: An information processing unit comprises: a first speech acquisition part which acquires first speech data on a first speech of a first speaker; a reproduction part which reproduces the first speech based upon the first speech data when a repetition mode for acquiring a speech repeating the first speech is set; a second speech acquisition part which acquires seconds speech data on a second speech of a second speaker as speech data on the repeated speech when the repetition mode is set; a recognition result acquisition part which acquires, based upon the first speech data and second speech data, first text information representing a first recognition result of the first speech and second text information representing a second recognition result of the second speech; and a recording generation part which generates, based upon the first text information and the second text information, recording data on a speech by text.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing apparatus, a program, and an information processing method.

Conventionally, there is known a technique of acquiring the voice of a speaker at a conference and creating minutes based on the voice acquired by using voice recognition technology.

In the minutes creation system disclosed in Patent Document 1 below, the voice of the speaker is recognized by using a preset dictionary, and the term that is not recognized as a result is recognized by the second dictionary. Output the request. Then, the minutes creation system receives the recognition result by the second dictionary and creates the minutes. According to such a minutes creation system, it is possible to improve the recognition accuracy because it is possible to make up for the place where the voice recognition by the preset dictionary could not be performed by the voice recognition by the second dictionary. Then, the work of the minutes creator to check and correct the created minutes can be reduced.

Japanese Unexamined Patent Publication No. 2017-191533

In the minutes creation system of Patent Document 1, if the voice of the speaker at the conference is difficult to recognize in the first place (for example, the volume is low), it is difficult to recognize even if it is recognized by the second dictionary. Therefore, there is a problem that the correction work of the minutes creator cannot be reduced.

Therefore, an object of the present invention is to provide an information processing device, a program, and an information processing method capable of accurately recording the content of an utterance even if the voice of the speaker is difficult to recognize.

The information processing device according to one aspect of the present invention has a first voice acquisition unit that acquires the first voice data of the first voice by the first speaker, and a repeat mode for acquiring the voice that repeats the first voice. When set, a playback unit that reproduces the first voice based on the first voice data, and when the repeat mode is set, the second voice by the second speaker is the voice data of the voice to be repeated. 2 The first text information indicating the first recognition result of the first voice and the second recognition of the second voice based on the second voice acquisition unit for acquiring the voice data, the first voice data and the second voice data. It includes a second text information indicating a result, a recognition result acquisition unit for acquiring, and a record generation unit for generating recorded data of a text-based speech based on the first text information and the second text information.

The program according to one aspect of the present invention has a first voice function for acquiring the first voice data of the first voice by the first speaker and a repeat mode for acquiring the voice for repeating the first voice in the computer. When set, a playback function that reproduces the first voice based on the first voice data, and when the repeat mode is set, the second voice by the second speaker is the voice data of the voice to be repeated. 2 The first text information indicating the first recognition result of the first voice and the second recognition of the second voice based on the second voice acquisition function for acquiring the voice data and the first voice data and the second voice data. A recognition result acquisition function for acquiring a second text information indicating a result, and a record generation function for generating recorded data of a text-based speech based on the first text information and the second text information are realized.

In the information processing method according to one aspect of the present invention, a repeat mode is set for the computer to acquire the first voice data of the first voice by the first speaker and to acquire the voice to repeat the first voice. In this case, the first voice is played based on the first voice data, and when the repeat mode is set, the second voice data of the second voice by the second speaker is acquired as the voice data of the voice to be repeated. , The first text information indicating the first recognition result of the first voice and the second text information indicating the second recognition result of the second voice are acquired based on the first voice data and the second voice data. , Generates textual speech recording data based on the first text information and the second text information.

According to the present invention, it is possible to provide an information processing device, a program, and an information processing method capable of accurately recording the content of an utterance even if the voice of the speaker is difficult to recognize.

It is a figure for demonstrating the system configuration example of the minutes making system which concerns on this embodiment. It is a figure for demonstrating the outline of the minutes making system which concerns on this embodiment. It is a figure which shows an example of the functional structure of the recording apparatus which concerns on this embodiment. It is a figure which shows an example of the screen example of the minutes making system which concerns on this embodiment. It is a figure which shows an example of the relationship between the recognition rate of the minutes making system which concerns on this embodiment and the volume, or the distance with a speaker. It is a figure which shows an example of the relationship between the power and the frequency of the minutes making system which concerns on this embodiment. It is a figure which shows the operation example of the recording apparatus which concerns on this embodiment. It is a figure which shows an example of the hardware composition of the recording apparatus which concerns on this embodiment.

A preferred embodiment of the present invention (hereinafter referred to as “the present embodiment”) will be described with reference to the accompanying drawings. In each figure, those with the same reference numerals have the same or similar configurations.

In the present embodiment, the "part", "means", "device", and "system" do not simply mean physical means, but the "part", "means", "device", and "system". Including the case where the function of is realized by software. Further, even if the functions of one "part", "means", "device", or "system" are realized by two or more physical means or devices, two or more "parts" or "means", The functions of "device" and "system" may be realized by one physical means or device.

<1. System configuration>
A system configuration example of the minutes creation system 1 according to the present embodiment will be described with reference to FIG. The minutes creation system 1 is a system that records the contents of utterances at a user's meeting or the like as minutes. However, the present invention is not limited to this. The present invention is applicable not only to the minutes but also to various systems for recording the content of the user's utterance. As shown in FIG. 1, the minutes creation system 1 includes a recording device 100 and a user terminal 200. Further, the minutes creation system 1 is connected to the voice recognition system 300 via the network N.

The network N is composed of a wireless network and a wired network. Examples of networks include mobile phone networks, PHS (Personal Handy-phone System) networks, wireless LAN (Local Area Network), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), and 5G (4th Generation). 5th Generation), WiMax®, infrared communication, Bluetooth®, wired LAN, telephone line, power line network, IEEE1394 compliant network, etc.

The recording device 100 is an information processing device capable of communicating with the user terminal 200 and the voice recognition system 300. The recording device 100 acquires the voices of the first speaker and the second speaker, which will be described later, and converts the acquired voice into text by voice recognition and records the voice.

The recording device 100 is a so-called smart speaker that responds to the acquired voice by dialogue or the like, but the purpose is not limited to this. As another example, the recording device 100 may be a general-purpose tablet terminal, a smartphone, or the like. The recording device 100 may use, for example, a tablet terminal or the like as the recording device 100 by installing a dedicated program on a general-purpose tablet terminal and executing this program.

The user terminal 200 is an information processing device such as a smartphone or a laptop terminal capable of inputting reception of a request from a user and communicating with the recording device 100. By executing a predetermined program, the user terminal 200 cooperates with the recording device 100 to display text data recorded by voice recognition (hereinafter, also referred to as “recorded data”), or edits the recorded data. Display a form to enable editing of text data.

The user includes the first speaker and the second speaker, and includes a person related to the minutes creation system 1 such as a person in charge of creating minutes in addition to the speaker.

The voice recognition system 300 is a system capable of communicating with the recording device 100. The voice recognition system 300 recognizes the user's voice based on the voice data (hereinafter, also simply referred to as “voice data”) indicating the user's voice received from the recording device 100.

<2. System overview>
The outline of the minutes preparation system 1 will be described with reference to FIG.

(1) As shown in FIG. 2, the first voice acquisition unit 131 of the recording device 100 acquires the first voice data of "starting a conference" as the first voice by the first speaker. (2) The recognition result acquisition unit 112 of the recording device 100 instructs the voice recognition system 300 to recognize the voice of the first voice based on the first voice data acquired in (1) above. (3) The recognition result acquisition unit 112 of the recording device 100 acquires the first text information indicating the first recognition result of the first voice from the voice recognition system 300.

The "first speaker" is a person who utters the first voice to be recorded by the minutes creation system 1. The first speaker may be, for example, a speaker at a conference.

(4) The reproduction unit 151 of the recording device 100 reproduces the first voice based on the first voice data when the repeat mode is set by the designation of the second speaker or the like. That is, the reproduction unit 151 outputs the first voice "start the conference". Here, the "repeated mode" is an operation mode for acquiring a voice (second voice) that repeats the first voice. Further, the repeat mode may be an operation mode in which the first voice is reproduced in order to acquire the second voice.

(5) When the repeat mode is set, when the second speaker repeats while listening to the first voice and utters the second voice "Start a meeting", the second voice acquisition unit 132 of the recording device 100 receives the second voice. The second voice data of the second voice to be repeated is acquired. (6) The recognition result acquisition unit 112 of the recording device 100 instructs the voice recognition system 300 to recognize the second voice based on the second voice data acquired in (5) above. (3) The recognition result acquisition unit 112 of the recording device 100 acquires the second text information indicating the second recognition result of the second voice from the voice recognition system 300.

The "second speaker" is a person who repeats the first voice and utters the second voice. When it is not necessary to distinguish between the "first speaker" and the "second speaker", they are also collectively referred to as "speaker" hereafter.

When it is not necessary to distinguish between the "first voice data" and the "second voice data", they are also collectively referred to as "voice data" hereafter.

When it is not necessary to distinguish between the "first recognition result" and the "second recognition result", they are also collectively referred to as "recognition result" hereafter.

When it is not necessary to distinguish between "first text information" and "second text information", they are also collectively referred to as "text information" hereafter.

(7) The comparison unit 113 of the recording device 100 compares the first text information and the second text information. (8) The record generation unit 114 of the recording device 100 generates textual recorded data as minutes based on the comparison result of the above (7) and the first text information and the second text information. The record generation unit 114 may, for example, replace the portion of the first text information that is inconsistent with the comparison result with the second text information with the second text information to generate the record data.

According to the above configuration, the recording device 100 uses text by comparison with the voice recognition result of the second voice that repeats the first voice even if the first voice of the first speaker in the conference is difficult to recognize. Recorded data can be generated as minutes. Therefore, according to the above configuration, the recording device 100 can accurately create the minutes.

<3. Function configuration>
The functional configuration of the recording apparatus 100 according to the present embodiment will be described with reference to FIG. As shown in FIG. 4, the recording device 100 includes a control unit 110, a voice acquisition unit 130, a communication unit 140, an output unit 150, and a storage unit 160.

The control unit 110 includes a recognition result acquisition unit 112 and a record generation unit 114. Further, the control unit 110 is, for example, a voice recognition unit 111, a comparison unit 113, an utterance data generation unit 115, a display unit 116, a reception unit 117, a reliability calculation unit 118, an estimation unit 119, an accuracy calculation unit 120, and voice data generation. A portion 121 or a processed portion 122 may be provided.

The control unit 110 sets the operation mode of the recording device 100 to the repeat mode based on the designation of the repeat mode received by the reception unit 117.

The voice recognition unit 111 recognizes the voice data of the speaker acquired by the voice acquisition unit 130. The voice recognition unit 111 generates text information indicating the result of this recognition. The voice recognition unit 111 may convert the voice data acquired by the voice acquisition unit 130 into text information by using the voice recognition technology, for example, in response to a voice recognition instruction from the recognition result acquisition unit 112. The voice recognition unit 111 may calculate its own recognition rate, for example.

The voice recognition unit 111 is based on, for example, the voice of untransmitted voice data when communication with the voice recognition system 300 becomes impossible while the communication unit 140 is transmitting voice data to the voice recognition system 300. May generate textual information.

The recognition result acquisition unit 112 acquires the first text information indicating the first recognition result of the first voice based on the first voice data acquired by the voice acquisition unit 130. Further, the recognition result acquisition unit 112 acquires the second text information indicating the second recognition result of the second voice based on the second voice data acquired by the voice acquisition unit 130. The recognition result acquisition unit 112 instructs the voice recognition system 300 or the voice recognition unit 111 to recognize the voice data based on the first voice data and the second voice data, for example. The recognition result acquisition unit 112 acquires the first text information and the second text information as a response to this instruction.

The recognition result acquisition unit 112 includes, for example, first text information and second text information divided into a plurality of sections based on the first utterance data and the second utterance data generated by the utterance data generation unit 115 described later. May be obtained.

The "section" is, for example, a silent section in which the voice level is zero or below a predetermined threshold in voice data (digital signal), and an utterance section (sound section) in which the voice level is higher than zero or exceeds a predetermined threshold. , May be included. Further, as another example, the section may be a range divided by a predetermined period.

The comparison unit 113 compares the first text information and the second text information. The comparison unit 113 may compare the first text information and the second text information for each of a plurality of sections, for example. As a comparison result, the comparison unit 113 outputs whether the first text information and the second text information match, or whether the first text information and the second text information do not match.

For example, the comparison unit 113 may compare which of the first text information and the second text information has the higher recognition accuracy (hereinafter, also simply referred to as “recognition accuracy”) for each of the plurality of utterance sections. The recognition accuracy may be, for example, the recognition rate in the voice recognition process. The comparison unit 113 may set this comparison result in the accuracy flag. Here, the "precision flag" is information indicating which of the first text information and the second text information has the higher recognition accuracy for each section. For the accuracy flag, for example, "1" is set for the one with relatively high recognition accuracy, and "0" is set for the other (the one with relatively low recognition accuracy).

The record generation unit 114 generates textual utterance record data based on the first text information and the second text information. The "recorded data of utterances" referred to here may be, for example, data that becomes the minutes of a meeting. The recorded data of the utterance is hereinafter simply referred to as "recorded data".

According to the above configuration, the recording generation unit 114 is, for example, a voice recognition result of the second voice that repeats the first voice even if the first voice of the first speaker in the conference is difficult to recognize. The first text information can be supplemented by using the second text information as well. Therefore, according to the above configuration, the record generation unit 114 can accurately generate the recorded data as minutes.

The record generation unit 114 may generate record data, for example, based on the comparison result by the comparison unit 113. The record generation unit 114 adopts, for example, the one with higher recognition accuracy in the comparison result by the comparison unit 113 with respect to the first text information and the second text information for each of a plurality of utterance sections based on the accuracy flag. The record generation unit 114 generates recorded data by combining the adopted first text information and second text information for each of a plurality of utterance sections.

According to the above configuration, the recording generation unit 114 can adopt the one having higher recognition accuracy of the first text information and the second text information as the recording data for each section. Therefore, the recording generation unit 114 can generate the recorded data with higher accuracy.

The record generation unit 114, for example, has the first text information and the second text information based on the selection of which of the first text information and the second text information received by the reception unit 117, which will be described later, is adopted as the utterance record. And may be combined for each section to generate recorded data.

According to the above configuration, the record generation unit 114 may display the comparison result to the user and generate the record data by combining the selected ones of the first text information or the second text information for each section. can. Therefore, according to the above configuration, since the user can select the first text information and the second text information, it is possible to generate the recorded data according to the user's request. Therefore, according to the above configuration, it is possible to improve the purpose and accuracy in preparing the minutes.

The record generation unit 114 is, for example, the text information received by the reception unit 117 for the section where the first text information and the second text information do not match in the comparison result by the comparison unit 113, and the first text information or the second text information. May be overwritten to generate recorded data.

According to the above configuration, since the record generation unit 114 can customize the record data with the text information edited by the user, it is possible to generate the record data more according to the user's request. Therefore, according to the above configuration, it is possible to improve the purpose and accuracy in recording the utterance.

The utterance data generation unit 115 generates a plurality of first utterance data corresponding to a plurality of sections of the first voice data. Further, the utterance data generation unit 115 generates a plurality of second utterance data corresponding to a plurality of sections of the second voice data. When there is no particular need to distinguish between the "first utterance data" and the "second utterance data", they are also collectively referred to as "utterance data" hereafter.

The utterance data generation unit 115 first detects a plurality of utterance sections and silent sections from the voice data. Next, the utterance data generation unit 115 divides the voice data into utterance data for each utterance section. In this way, the utterance data generation unit 115 generates a plurality of utterance data corresponding to the plurality of utterance sections of the voice data.

The display unit 116 causes the user terminal 200 to display the comparison result by the comparison unit 113. The display unit 116 generates display information, for example, as a comparison result. The display information includes, for example, recorded data and accuracy flags for each of the plurality of sections. This "recorded data of each of the plurality of sections" includes the first text information of each of the plurality of sections and the second text information of each of the plurality of sections. The display information is also information for displaying the screens of the minutes creation system 1 such as the first comparison screen A1 and the second comparison screen A2 shown in FIG. 4 on the user terminal 200. The display unit 116 transmits the generated display information to the user terminal 200 via the communication unit 140.

The display unit 116 may display, for example, the reliability of each of the first recognition result and the second recognition result on the user terminal 200 together with the comparison result by the comparison unit 113. Here, the "reliability" is the degree of certainty (confidence) of the voice recognition result. The displayed information includes these reliabilitys.

According to the above configuration, the display unit 116 can cause the user to display the reliability of each recognition result together with the comparison result between the first text information and the second text information. Therefore, according to the above configuration, the user can confirm the reliability that can be an index when selecting which of the first text information and the second text information is to be adopted. Therefore, the display unit 116 can improve usability in selecting the first text information and the second text information.

The display unit 116 may display, for example, the recognition accuracy of each of the first recognition result and the second recognition result on the user terminal 200 together with the comparison result by the comparison unit 113. The display information includes these recognition accuracy.

According to the above configuration, the display unit 116 can cause the user to display the recognition accuracy of each recognition result together with the comparison result between the first text information and the second text information. Therefore, according to the above configuration, the user can confirm the reliability that can be an index when selecting which of the first text information and the second text information is to be adopted. Therefore, the display unit 116 can improve usability in selecting the first text information and the second text information.

The display unit 116 is, for example, a user terminal for editing a section for editing the first text information or the second text information of the section in which the first text information and the second text information do not match in the comparison result. It may be displayed at 200. The display information includes this edit form.

The reception unit 117 records either the first recognition result (first text information) or the second recognition result (second text information) of the first speaker's utterance from the user terminal 200 for each of the plurality of sections. Accepts the choice of whether to adopt as. The reception unit 117 uses, for example, either the first text information or the second text information of the first speaker for the section in which the first text information and the second text information displayed by the display unit 116 do not match. You may accept the choice of adopting it as the minutes of the utterance.

The reception unit 117 may receive, for example, the text information input by the user for the edit form displayed by the display unit 116 from the user terminal 200.

Here, an example of the comparison screen displayed by the display unit 116 will be described with reference to FIG. FIG. 4A shows an example of a first comparison screen that displays the first text information divided for each utterance section as the first recognition result. FIG. 4B shows an example of a second comparison screen that displays the second text information divided for each utterance section as the second recognition result. In this example, in order to facilitate the explanation, an example in which the first comparison screen and the second comparison screen are displayed as separate screens will be described, but the present invention is not limited to this. The display contents of the first comparison screen and the display contents of the second comparison screen may be displayed side by side on one screen, for example.

As shown in FIG. 4A, the display unit 116 causes the user terminal 200 to display the first comparison screen A1. The first comparison screen A1 displays a first voice data display area a11 and a play button a12 for playing back the voice of the utterance data displayed in the first voice data display area a11 and the second voice data display area a21. -Includes a save button a13 for saving the edited text information as recorded data.

The first voice data display area a11 includes a plurality of first utterance data display areas for displaying each first utterance data divided for each utterance section. In this example, the first utterance data display area a111 and the first utterance data display area a112 will be described from among the plurality of first utterance data display areas. The first utterance data display area a111 displays the first text information "hello" recognizing the voice generated by the participant 1 with the first utterance as the "participant 1". Further, the first utterance data display area a112 displays the first text information "kochi" recognizing the voice generated by the participant 2, with the first utterance as the "participant 2".

As shown in FIG. 4B, the display unit 116 causes the user terminal 200 to display the second comparison screen A2. The second comparison screen A2 includes a second audio data display area a21, a play button a12, and a save button a13.

The second voice data display area a21 includes a plurality of second utterance data display areas, like the first voice data display area a11.

In this example, from the plurality of second utterance data display areas, the second utterance data display area a211 corresponding to the first utterance data display area a111 and the second utterance data display corresponding to the first utterance data display area a112. The area a212 will be described with reference to the area a212. With respect to the first utterance data displayed in the first utterance data display area a111, the second utterance data of the second voice that repeats the first voice is displayed in the second utterance data display area a112. With respect to the first utterance data displayed in the first utterance data display area a112, the second utterance data of the second voice that repeats the first voice is displayed in the second utterance data display area a212.

In the second utterance data display area a211, the second speaker is set as "reciter 1", and the second text information "hello" that recognizes the voice generated by the repeater 1 is displayed. Further, in the second utterance data display area a212, the second utterance person is also regarded as the "reciter 1", and the second text information "hello" recognizing the voice generated by the repeater 1 is displayed.

In the first utterance data display area a111 and the second utterance data display area a211, since the first text information and the second text information in the corresponding utterance section match, the respective recognition results are displayed. When the user presses (tap operation, click operation, etc.) any of the utterance data display areas that match in this way, the display unit 116 edits the text information in this utterance data display area. The input window a14 (one aspect of the editing form) may be displayed on the user terminal 200.

The display unit 116 can change the display mode of the utterance data display area regarding the mismatched portion so that the mismatched portion between the first text information and the second text information can be seen at a glance so as to be different from the matched portion. Specifically, since the first utterance data display area a112 and the second utterance data display area a212 do not match the first text information and the second text information in the corresponding utterance section, their respective recognition results are displayed. In addition, display comparison results and edit forms. More specifically, in the first utterance data display area a112, as a comparison result, an icon of a troubled face indicating that the recognition accuracy is relatively low, and text information in which the character color (for example, red) and the font are changed are used. , Is displayed. The second utterance data display area a212 displays, as a comparison result, a smile icon indicating that the recognition accuracy is relatively high, and text information in which the character color (for example, black) or the font is changed. Further, the second utterance data display area a212 is also a text form (one aspect of the editing form) in which the user can directly edit the text information. Further, this text form may be a display mode indicating that editing by the user is unnecessary when the recognition accuracy of the second recognition result is higher than a predetermined threshold value (for example, the background color (for example, gray) may be changed. May be). When the user inputs text information in this text form, the reception unit 117 accepts the input text information.

When the recognition accuracy of the second recognition result is higher than the recognition accuracy of the first recognition result for the mismatched portion between the first text information and the second text information, the display unit 116 sets the second utterance data as in the above example. Display the display area a212 as a text form. On the other hand, when the recognition accuracy of the first recognition result is higher than the recognition accuracy of the second recognition result, the display unit 116 copies the first text information of the first utterance data display area a112 to the second utterance data display area a212. After (overwriting), the second utterance data display area a212 may be displayed as a text form. At this time, the display unit 116 may display the second text information of the second recognition result in the remarks area (not shown) of the second utterance data display area a212.

As described above, the user can select which of the first utterance data display area a112 and the second utterance data display area a212, which are the mismatched parts, to be adopted as the recorded data by pressing each area. .. When the user makes a selection, the reception unit 117 accepts this selection.

According to the above configuration, the display unit 116 can display the first text information and the second text information in a comparable manner for each section by the first comparison screen A1 and the second comparison screen A2. Further, according to the above configuration, the display unit 116 can change the display mode so that the mismatched portion between the first text information and the second text information can be seen at a glance. Therefore, according to the above configuration, the display unit 116 can improve usability in the UI such as confirmation of text information and selection of text information.

The explanation will be continued by returning to FIG. The reception unit 117 may, for example, receive a designation from the user terminal 200 to switch the audio of the left and right channels in the stereophonic reproduction by the reproduction unit.

The reception unit 117 may accept the designation of the repeat mode from the user terminal 200, for example.

The reliability calculation unit 118 calculates the reliability of each of the first recognition result and the second recognition result. For example, the reliability calculation unit 118 may calculate the reliability of each word included in the recognition result, aggregate the calculated reliability of each word, and calculate the reliability of the recognition result.

The word-by-word confidence may have, for example, a value in a predetermined range (eg, in the range of 0.0 to 1.0). The closer the number is 1.0, or the upper limit, within this predetermined range of values, the less the word-by-word confidence is that there are relatively few other competitors with scores similar to that word. On the other hand, the closer the number is 0.0, that is, the lower limit, in this predetermined range of values, the more the word-by-word confidence indicates that there are relatively many other competitors with scores similar to that word. .. That is, the closer the numerical value is to the upper limit within the predetermined range, the more the reliability of each word is that there is no other candidate close to the word of the first candidate in the recognition result, and the recognition result is output with confidence (confidence). It can be said that it was done.

There are several possible methods for calculating the reliability of words. For example, Komatani and Kawahara, "Dialogue processing for efficient confirmation and guidance using the reliability of speech recognition results" (IPSJ Paper) Journal, Vol. 43, No. 10, pp3078-3086) is known.

The estimation unit 119 estimates the first distance between the first speaker and the recording device 100 based on the first voice data. Further, the estimation unit 119 estimates the second distance between the second speaker and the recording device 100 based on the second voice data. Here, the "distance between the speaker and the recording device 100 (hereinafter, also simply referred to as" the distance between the speaker ")" is specifically defined as a plurality of microphones (microphones) of the speaker and the voice input device 817. It may be the distance between the array) (hereinafter, also simply referred to as “microphone”).

The estimation unit 119 estimates, for example, the direction and position of the speaker or the distance to the speaker based on the voice data of each speaker. The estimation unit 119 may record the estimation result (direction and position of the speaker, distance from the speaker, etc.) in the storage unit 160 as position information. For example, the estimation unit 119 calculates a cross-correlation function between the time waveforms of two voice signals input to the voice input device 817, and calculates the arrival time difference of the sound from the calculated cross-correlation function. The estimation unit 119 may estimate the direction, position, or distance of the speaker based on the calculated sound arrival time difference.

The accuracy calculation unit 120 respectively of the first recognition result and the second recognition result based on the combination of the first volume of the first voice and the second volume of the second voice (hereinafter, also referred to as “volume combination”). Calculate recognition accuracy. Further, the accuracy calculation unit 120 calculates the recognition accuracy of each of the first recognition result and the second recognition result based on the combination of the first distance and the second distance (hereinafter, also referred to as “distance combination”). The accuracy calculation unit 120 may calculate the recognition accuracy of each of the first recognition result and the second recognition result based on at least one of the combination of volume and the combination of distance.

For example, when the voice recognition by the voice recognition system 300 is used, the accuracy calculation unit 120 may acquire the recognition rate together with the recognition result from the voice recognition system 300. For example, when using the voice recognition by the voice recognition unit 111, the accuracy calculation unit 120 may acquire the recognition rate together with the recognition result from the voice recognition unit 111.

For example, in the combination of volumes, the accuracy calculation unit 120 inputs the volume of the voice in a predetermined learning period and the recognition rate of the corresponding voice as learning data, so that the volume and the recognition rate as shown in FIG. 5A are obtained. The first pattern model of may be constructed. For example, the accuracy calculation unit 120 may construct a first pattern model by using statistical processing by regression analysis with the volume as an explanatory variable (feature amount) and the recognition rate as an objective variable (feature amount). The accuracy calculation unit 120 may input the volume of voice into the constructed first pattern model and calculate the recognition rate. For example, the accuracy calculation unit 120 divides the range in which the volume can be taken into three stages (“high”, “medium”, and “low”). The accuracy calculation unit 120 calculates, for example, the recognition rate of the volume belonging to the predetermined range (R1) among the three divided ranges as “high”.

For example, in the combination of distances, the accuracy calculation unit 120 inputs the distance between the speaker and the microphone in a predetermined learning period and the corresponding recognition rate as learning data, so that the speaker as shown in FIG. 5 (b). A second pattern model of the distance to and the recognition rate is constructed. The accuracy calculation unit 120 may construct a second pattern model by using statistical processing by regression analysis, for example, using the distance to the speaker as an explanatory variable (feature amount) and the recognition rate as an objective variable (feature amount). .. The accuracy calculation unit 120 may input the distance to the speaker into the constructed second pattern model and calculate the recognition rate. For example, the accuracy calculation unit 120 divides the range in which the distance from the speaker can be taken into three stages (“high”, “medium”, and “low”). The accuracy calculation unit 120 calculates, for example, the recognition rate of the distance to the speaker belonging to the predetermined range (R2) among the three set ranges as “high”.

The accuracy calculation unit 120 may calculate the recognition accuracy of each of the first recognition result and the second recognition result, for example, based on the combination of the frequency of the first voice and the frequency of the second voice. The accuracy calculation unit 120 calculates, for example, a statistical value (average value or median value) or frequency band of the frequency of the first voice or the second voice for each speech section, and the lower limit of the statistical value or the frequency band is a predetermined threshold value. If it is higher, the recognition rate in this speech section may be calculated as "high". That is, the accuracy calculation unit 120 may calculate the recognition rate high when the voice contains a large amount of high frequency components.

For example, the accuracy calculation unit 120 may extract a consonant or at least one of the powers (or sound pressure levels) in a high frequency range above a predetermined threshold value as a feature amount contained in the voice. The "power" here is so-called acoustic power, which indicates the weight (power) for each frequency in the frequency analysis of sound, and is different from the loudness and intensity (volume) of the sound perceived by human hearing. The power is the strength of a consonant or a voice in a high frequency range above a predetermined threshold. The accuracy calculation unit 120 may calculate the recognition rate based on the extracted feature amount. For example, the accuracy calculation unit 120 weights the recognition rate calculated above by the power of the consonant, and the weighted recognition rate has the above three stages (“high”, “medium”, and “low”). You may calculate with.

The accuracy calculation unit 120 plots, for example, the sound pressure level and frequency of voice as shown in FIG. The accuracy calculation unit 120 divides the range that the plotted data can take into three recognition rate stages (“high”, “medium”, and “low”). The accuracy calculation unit 120 may calculate the recognition rate depending on which of the three areas the sound pressure level and frequency of the voice belong to, for example.

The accuracy calculation unit 120 is, for example, at least one of (a) volume, (b) distance from the speaker, (c) frequency, (d) consonant or power in a high frequency range above a predetermined threshold, as described above. Based on the combination of the recognition rate calculated by the above and (e) the recognition rate acquired from the voice recognition system 300 and the voice recognition unit 111, a composite recognition rate (hereinafter, also referred to as "composite recognition rate") is obtained. You may calculate.

The accuracy calculation unit 120 may, for example, calculate the weighted average of each of the above recognition rates (a) to (e) and calculate the calculated weighted average as the compound recognition rate. For example, the accuracy calculation unit 120 may set the importance of the above (a) and (b) to be higher than the other (c) to (e) in this weighted average. The accuracy calculation unit 120 may, for example, weight a coefficient proportional to this importance by multiplying each recognition rate. Specifically, the accuracy calculation unit 120 may calculate the compound recognition rate by the following formula.

Combined recognition rate = (α x recognition rate of (e) above + β x recognition rate of (a) above + θ x recognition rate of (a) above + δ, recognition rate of (c) above) / (α + β + θ + δ)

“Α” is the weighting coefficient of (e) above, “β” is the weighting coefficient of (a) above, that is, the volume, and “θ” is the weighting coefficient of (b), that is, the distance. “Δ” is the above (c), that is, the frequency weighting coefficient. β and θ may be larger than α and δ depending on the set importance.

The voice data generation unit 121 generates a third voice data for outputting the third voice based on the first text information by using the voice synthesis process. The third voice may be, for example, a voice that reads out the character string of the first text information.

The voice data generation unit 121 may generate response information based on, for example, text information. Here, the "response information" is information for the recording device 100 to respond to the voice of the user. The voice data generation unit 121 analyzes text information using, for example, natural language processing. Then, the voice data generation unit 121 identifies the content of the response to the user's voice by this analysis, and generates the response information. The voice data generation unit 121 may generate voice data for a response to the user's voice based on the response information by using the voice synthesis process.

The voice data generation unit 121 extracts, for example, the words "minutes" and "start" by morphologically analyzing the content "start minutes" of the user's voice. Next, the voice data generation unit 121 searches the dictionary information using these extracted words as a search key and identifies the content of the corresponding response. The content of this response is (a) execution of a series of processes such as acquisition of the first voice data and voice recognition processing for creating the minutes of the utterance of the first speaker, and (b) "minutes" to the user. Is executed, and the process of outputting the voice is executed.

"Dictionary information" is information that associates a word or a combination of a plurality of words with the content of a response. The dictionary information associates, for example, the combination of the words "minutes" and "start" with the content of the response to execute the processes (a) and (b) above.

The processing unit 122 processes the first audio data and the third audio data into stereo audio data. Here, the "stereo audio data" is audio data for stereophonically reproducing any two audios of the first audio, the second audio, and the third audio.

For example, the processing unit 122 may process at least two of the first audio data, the second audio data, and the third audio data into stereo audio data. In this case, the stereo audio data is (A) a combination of the first audio data and the second audio data, (B) a combination of the first audio data and the third audio data, and (C) the second audio data and the third. It may be any one of the three patterns to be combined with the voice data.

For example, the processing unit 122 may localize the audio sound image of the first audio data, the second audio data, or the third audio data as preprocessing for processing the stereo audio data. For example, the processing unit 122 may localize the sound image of the first voice at the position of the virtual sound source for each first speaker (channel) for the first voice data. For example, if the position (angle) of the speaker is biased, the position of this virtual sound source becomes a voice that is difficult to hear, that is, a voice that is difficult to recognize. Therefore, even if the position of the speaker is set to be even. good.

The voice acquisition unit 130 acquires voice data of the user's voice. The voice acquisition unit 130 includes a first voice acquisition unit 131 and a second voice acquisition unit 132. The first voice acquisition unit 131 acquires the first voice data of the first voice by the first speaker. The second voice acquisition unit 132 acquires the second voice data of the second voice by the second speaker as the voice data of the voice to be repeated when the repeat mode is set.

For example, when acquiring voice data for each speaker, the voice acquisition unit 130 may perform directivity processing or sound source separation processing for separating the sound source with respect to the voice signal input to the voice input device 817. The directional processing is, for example, signal processing (beamforming processing) that emphasizes the sound from the direction of the speaker and suppresses the sound from a direction other than the speaker. The sound source separation process is a process of extracting target sounds for each direction of the speaker and separating them individually. The voice acquisition unit 130 separates the speakers, and acquires voice data of voice from each speaker direction for each separated speaker.

The voice acquisition unit 130 may, for example, use a microphone having directivity for each speaker to identify the microphone that collects the sound, identify the speaker, and acquire voice data for each identified speaker.

The communication unit 140 transmits / receives various information / data such as voice data and text information to / from the user terminal 200, the voice recognition system 300, and the like via the network N.

The output unit 150 outputs a response to the voice based on the response information. The output mode of the output unit 150 may be any mode. The output mode of the output unit 150 may be, for example, audio output, screen output, file output, message output, or the like. The output unit 150 includes a reproduction unit 151.

The reproduction unit 151 reproduces the first voice based on the first voice data when the repeat mode is set.

The reproduction unit 151 may, for example, stereophonically reproduce at least two of the first sound, the second sound, and the third sound based on the stereo sound data processed by the processing unit. Further, in the stereophonic reproduction, the reproduction unit 151 may exchange the audio of the left and right channels based on the designation of the exchange of the left and right channels accepted by the reception unit 117. For example, when the left channel is the first sound and the right channel is the third sound, the playback unit 151 has the left channel as the third sound and the right channel as the third sound based on the above-mentioned specification of exchanging the left and right channels. It may be replaced with the first sound and played back stereophonically.

For example, when the repeat mode is set based on the stereo audio data processed by the processing unit, the reproduction unit 151 stereophonically plays the first sound and the third sound instead of playing the first sound. You may play it.

According to the above configuration, the reproduction unit 151 can make the second speaker recognize the difference between the first voice and the third voice by stereophonically playing the first voice and the third voice. Therefore, according to the above configuration, the reproduction unit 151 can make the second speaker repeat the first voice while recognizing the difference between the first voice and the third voice that recognizes the first voice.

The storage unit 160 stores audio data (including stereo audio data). Further, the storage unit 160, for example, associates with the voice data, text information indicating the recognition result of the voice data, position information of the speaker regarding the voice data, accuracy information indicating the recognition accuracy of the recognition result of the voice data, and recognition of the voice data. The reliability information indicating the reliability of the result and / or the response information to the recognition result of the voice data may be stored. Further, the storage unit 160 may store dictionary information, for example.

The storage unit 160 may use a database management system (DBMS) to store the above information, or may use a file system to store the above information. When using the DBMS, a table may be provided for each of the above information, and the tables may be associated with each other to manage the information.

<4. Operation example>
An operation example of the recording device 100 will be described with reference to FIG. 7. The order of processing of the operation example of FIG. 7 shown below is an example, and may be changed as appropriate.

As shown in FIG. 7, the first voice acquisition unit 131 of the recording device 100 acquires the first voice data of the first voice by the first speaker (S10). Next, the control unit 110 sets the repeat mode based on the designation of the repeat mode received from the user terminal 200 (S11).

Next, the reproduction unit 151 reproduces the first voice based on the first voice data when the repeat mode for acquiring the voice to repeat the first voice is set (S12). Next, when the repeat mode is set, the second voice acquisition unit 132 acquires the second voice data of the second voice by the second speaker as the voice data of the voice to be repeated (S13).

Next, the recognition result acquisition unit 112 shows the first text information indicating the first recognition result of the first voice and the second recognition result of the second voice based on the first voice data and the second voice data. The second text information and the second text information are acquired (S15).

Next, the comparison unit 113 compares the first text information and the second text information (S16). The record generation unit 114 generates textual utterance record data based on the comparison result of the first text information, the second text information, and the comparison unit 113 (S17).

<5. Hardware configuration>
With reference to FIG. 8, an example of the hardware configuration in the case where the recording device 100 described above is realized by the computer 800 will be described. The function of each device can be realized by dividing it into a plurality of devices.

As shown in FIG. 8, the computer 800 includes a processor 801 and a memory 803, a storage device 805, an input I / F unit 807, a data I / F unit 809, a communication I / F unit 811 and a display device 813. , Includes audio input device 817 and audio output device 819.

The processor 801 controls various processes in the computer 800 by executing a program stored in the memory 803. For example, each functional unit included in the control unit 110 of the recording device 100 can be realized by the processor 801 executing a program temporarily stored in the memory 803.

The memory 803 is a storage medium such as, for example, a RAM (Random Access Memory). The memory 803 temporarily stores the program code of the program executed by the processor 801 and the data required when the program is executed.

The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or a flash memory. The storage device 805 stores an operating system and various programs for realizing each of the above configurations. In addition, the storage device 805 can also store a table for registering voice data, text information, position information, accuracy information, reliability information, response information, and the like, and a DB for managing this table. Such programs and data are referenced from the processor 801 by being loaded into the memory 803 as needed.

The input I / F unit 807 is a device for receiving input from the user. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, a wearable device, and the like. The input I / F unit 807 may be connected to the computer 800 via an interface such as USB (Universal Serial Bus).

The data I / F unit 809 is a device for inputting data from the outside of the computer 800. Specific examples of the data I / F unit 809 include a drive device for reading data stored in various storage media. It is also conceivable that the data I / F unit 809 is provided outside the computer 800. In that case, the data I / F unit 809 is connected to the computer 800 via an interface such as USB.

The communication I / F unit 811 is a device for performing data communication via the Internet N by wire or wirelessly with an external device of the computer 800. It is also conceivable that the communication I / F unit 811 is provided outside the computer 800. In that case, the communication I / F unit 811 is connected to the computer 800 via an interface such as USB.

The display device 813 is a device for displaying various information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminence) display, a display of a wearable device, and the like. The display device 813 may be provided outside the computer 800. In that case, the display device 813 is connected to the computer 800 via, for example, a display cable or the like. Further, when the touch panel is adopted as the input I / F unit 807, the display device 813 can be integrally configured with the input I / F unit 807.

The voice input device 817 is an input device for detecting voice such as a microphone. The voice input device 817 is, for example, a single or a plurality of microphones (microphone arrays) that acquire an analog vibration signal including a voice signal, an amplifier that amplifies the analog vibration signal, and an A / D that converts the analog vibration signal into a digital signal. It is equipped with a conversion unit and the like. The voice input device 817 detects, for example, the voice emitted by the user.

The audio output device 819 is an output device for outputting audio, and is, for example, a speaker or the like. Further, the audio output device 819 may be a device for reproducing sound in stereo on headphones or earphones.

It should be noted that the present embodiment is an example for explaining the present invention, and is not intended to limit the present invention to only the embodiment thereof. Further, the present invention can be modified in various ways as long as it does not deviate from the gist thereof. Further, those skilled in the art can adopt an embodiment in which each element described below is replaced with an equal one, and such an embodiment is also included in the scope of the present invention.

[Modification example]
Although the present invention has been described based on the above embodiment, the following cases are also included in the present invention.

[Modification 1]
At least a part of each configuration in the recording device 100 according to the above embodiment may be provided with a program dedicated to the minutes creation system 1 mounted on the user terminal 200 or the server device (not shown). For example, this program is provided with each function unit of the control unit 110 of the recording device 100 and the voice acquisition unit 130, and the output unit 150 uses the function provided as standard in the user terminal 200 in the user terminal 200. All may be realized. Further, at this time, among the functional units of the control unit 110, the reliability calculation unit 118, the estimation unit 119, the accuracy calculation unit 120, etc., which have a relatively high processing load, may be mounted on the server device. The user terminal 200 may only receive processing instructions for these functions of the server device and processing results for the instructions.

[Modification 2]
In the above embodiment, an example is shown in which the user is allowed to select which text information is to be adopted as the record of the utterance of the first speaker, that is, the minutes of the section where the first text information and the second text information do not match. However, it is not limited to this. In the minutes preparation system 1, which one may be automatically selected may be selected based on the recognition accuracy of each of the first recognition result and the second recognition result.

The control unit 110 includes a selection unit (not shown). The selection unit determines the first text information and the second text information based on at least either the reliability or the recognition accuracy of the section in which the first text information and the second text information do not match in the comparison result by the comparison unit 113. Which of the above is adopted as the record of the utterance of the first speaker is selected.
For example, the selection unit may select the text information having the higher reliability as the record of the utterance of the first speaker.

The record generation unit 114 may generate record data by combining the first text information and the second text information for each section based on the selection result by the selection unit, for example.

According to the above configuration, the user can save the trouble of selecting either the first text information or the second text information, so that the utterance record can be efficiently generated.

[Modification 3]
Although not shown in the above embodiment, when the repeat mode is set, when the second speaker is reciting, the accuracy calculation unit 120 is the second voice data acquired by the second voice acquisition unit 132. 2 The voice recognition accuracy may be calculated at any time. Then, the display unit 116 may display the calculated recognition accuracy on the user terminal 200 at any time. According to such a configuration, the display unit 116 can display the recognition accuracy of the second voice in a timely manner when the second speaker is reciting. Therefore, according to the above configuration, the second speaker can repeat, for example, while reviewing so that his / her own volume and the distance from the recording device 100 can be recognized more accurately.

1 ... Minutes creation system, 100 ... Recording device, 110 ... Control unit, 111 ... Voice recognition unit, 112 ... Recognition result acquisition unit, 113 ... Comparison unit, 114 ... Record generation unit, 115 ... Speech data generation unit, 116 ... Display unit, 117 ... reception unit, 118 ... reliability calculation unit, 119 ... estimation unit, 120 ... accuracy calculation unit, 121 ... voice data generation unit, 122 ... processing unit, 130 ... voice acquisition unit, 131 ... first voice acquisition Unit, 132 ... Second voice acquisition unit, 140 ... Communication unit, 150 ... Output unit, 151 ... Playback unit, 160 ... Storage unit, 200 ... User terminal, 300 ... Voice recognition system, 800 ... Computer, 801 ... Processor, 803 ... Memory, 805 ... Storage device, 807 ... Input I / F unit, 809 ... Data I / F unit, 811 ... Communication I / F unit, 813 ... Display device, 817 ... Voice input device, 819 ... Voice output device.

Claims (10)

The first voice acquisition unit that acquires the first voice data of the first voice by the first speaker, and
A reproduction unit that reproduces the first voice based on the first voice data when a repeat mode for acquiring the voice that repeats the first voice is set.
A second voice acquisition unit that acquires the second voice data of the second voice by the second speaker as the voice data of the voice to be repeated when the repeat mode is set.
Based on the first voice data and the second voice data, the first text information indicating the first recognition result of the first voice, the second text information indicating the second recognition result of the second voice, and the second text information. The recognition result acquisition unit to acquire
A record generation unit that generates recorded data of the utterance of the first speaker by text based on the first text information and the second text information is provided.
Information processing equipment. An utterance data generation unit that generates a plurality of first utterance data corresponding to a plurality of sections of the first voice data and a plurality of second utterance data corresponding to a plurality of sections of the second voice data from the voice data. , Further prepared,
The recognition result acquisition unit acquires the first text information and the second text information divided into each of the plurality of sections based on the first utterance data and the second utterance data.
The information processing apparatus further includes a comparison unit for comparing the first text information and the second text information for each of the plurality of sections.
The record generation unit generates the record data based on the comparison result by the comparison unit.
The information processing apparatus according to claim 1. A display unit that displays the comparison result on the user terminal of the user,
A reception unit that accepts from the user terminal the selection of whether to adopt the first text information or the second text information as a record of the utterance of the first speaker for each of the plurality of sections.
The record generation unit further combines the first text information and the second text information for each section based on the selection accepted by the reception unit to generate the record data.
The information processing apparatus according to claim 2. Further, a reliability calculation unit for calculating the reliability of each of the first recognition result and the second recognition result is provided.
The display unit causes the user terminal to display the reliability together with the comparison result.
The information processing apparatus according to claim 3. Based on the first voice data and the second voice data, the first distance between the first speaker and the information processing device and the second distance between the second speaker and the information processing device. 2 distances, an estimation unit that estimates, and
The first recognition result and the first recognition result based on at least one combination of the first volume of the first voice and the second volume of the second voice, or the combination of the first distance and the second distance. 2 Further equipped with an accuracy calculation unit that calculates the recognition accuracy of each recognition result.
The display unit causes the user terminal to display the recognition accuracy together with the comparison result.
The information processing apparatus according to claim 3 or 4. The display unit provides an edit form for editing the first text information or the second text information of the section for the section in which the first text information and the second text information do not match in the comparison result. Display it on the user terminal
The reception unit receives the text information input by the user for the edit form from the user terminal, and receives the text information.
The record generation unit generates the record data by overwriting the first text information or the second text information with the text information received by the reception unit for the inconsistent section.
The information processing apparatus according to claim 5. An estimation unit that estimates a first distance between the first speaker and the information processing device and a second distance between the second speaker and the information processing device.
The first recognition result and the first recognition result based on at least one combination of the first volume of the first voice and the second volume of the second voice, or the combination of the first distance and the second distance. 2 The accuracy calculation unit that calculates the recognition accuracy of each recognition result,
A reliability calculation unit that calculates the reliability of each of the first recognition result and the second recognition result,
With respect to the section in which the first text information and the second text information do not match in the comparison result, the first text information and the second text information are based on at least one of the reliability or the recognition accuracy. A selection unit that selects which one is to be adopted as the record of the utterance of the first speaker, and
The record generation unit further combines the first text information and the second text information for each section based on the selection result by the selection unit to generate the record data.
The information processing apparatus according to claim 2. A voice data generation unit that generates a third voice data for outputting a third voice based on the first text information by using a voice synthesis process.
A processing unit that processes the first audio data and the third audio data into stereo audio data for stereophonically reproducing the first audio and the third audio.
When the repeat mode is set, the reproduction unit performs the stereophonic reproduction instead of the reproduction of the first audio based on the stereo audio data.
The information processing apparatus according to any one of claims 1 to 7. On the computer
The first voice function to acquire the first voice data of the first voice by the first speaker, and
A reproduction function for reproducing the first voice based on the first voice data when a repeat mode for acquiring the voice for reciting the first voice is set.
A second voice acquisition function that acquires the second voice data of the second voice by the second speaker as the voice data of the voice to be repeated when the repeat mode is set.
Based on the first voice data and the second voice data, the first text information indicating the first recognition result of the first voice, the second text information indicating the second recognition result of the second voice, and the second text information. With the recognition result acquisition function to acquire
A record generation function for generating recorded data of the utterance of the first speaker by text based on the first text information and the second text information is realized.
program. The computer
Acquire the first voice data of the first voice by the first speaker,
When the repeat mode for acquiring the voice to repeat the first voice is set, the first voice is reproduced based on the first voice data.
When the repeat mode is set, the second voice data of the second voice by the second speaker is acquired as the voice data of the voice to be repeated.
Based on the first voice data and the second voice data, the first text information indicating the first recognition result of the first voice, the second text information indicating the second recognition result of the second voice, and the second text information. To get,
Based on the first text information and the second text information, the recorded data of the utterance of the first speaker by text is generated.
Information processing method.

Images