JP2019050482A - Information acquisition device, display method, and program - Google Patents

Abstract

To provide an information acquisition device, a display method and a program, capable of intuitively grasping a position of a speaker during recording.SOLUTION: An information acquisition device 2 includes: a display unit 23 which can display an image; a first voice collection section 20 and a second voice collection section 21 installed at positions different from each other and collecting voice emitted from each of a plurality of sound sources and creating voice data; a sound source position estimation section 295 for estimating positions of the plurality of sound sources on the basis of sound data generated by each of the first voice collection section 20 and the second voice selection 21; and a display control section 303 for displaying sound source position information concerning positions of each of the plurality of sound sources on a display unit 23 on the basis of an estimation result estimated by the sound source position estimation section 295.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information acquisition device, display method and program for estimating and displaying the position of each sound source based on voice data.

  In recent years, a technique for specifying the position of a sound source using a plurality of microphone arrays is known (see Patent Document 1). In this technique, a space centered on a point determined in relation to the position of the microphone array based on each of the plurality of sound source signals obtained from the outputs of the plurality of microphone arrays and the positional relationship of each of the plurality of microphone arrays The MUSIC power is calculated at predetermined time intervals for each of a plurality of directions defined therein, and the peak of the MUSIC power is identified as the sound source position, and then the audio signal of the sound source position is separated from the output signal of the microphone array.

JP, 2012-211768, A

  By the way, when the user prepares a deduction such as a meeting based on voice data, there are times when it is desired to grasp the position of the speaker during recording. However, in Patent Document 1 described above, only the sound source position is specified. Therefore, when the user prepares a deduction such as a meeting based on voice data, the position of the speaker at the time of recording can be intuitively grasped. There was a problem that it was impossible.

  The present invention has been made in view of the above, and it is an object of the present invention to provide an information acquiring apparatus, a display method and a program which can intuitively grasp the position of a speaker at the time of recording.

  In order to solve the problems described above and achieve the object, the information acquisition device according to the present invention is provided at a position different from each other from a display unit capable of displaying an image, and emitted from each of a plurality of sound sources. A plurality of sound collection units that collect sound and generate sound data; and a sound source position estimation unit that estimates positions of the plurality of sound sources based on the sound data generated by each of the plurality of sound collection units; A display control unit configured to display, on the display unit, sound source position information on each of the positions of the plurality of sound sources based on the estimation result estimated by the sound source position estimating unit.

  In the display method according to the present invention, a display unit capable of displaying an image and a plurality of display units provided at mutually different positions and generating voice data by collecting voices emitted from each of a plurality of sound sources. A display method executed by an information acquisition apparatus including a sound collection unit, wherein sound source position estimation is performed to estimate the positions of the plurality of sound sources based on the voice data generated by each of the plurality of sound collection units The display control step may include the steps of: displaying, on the display unit, sound source position information on each position of the plurality of sound sources based on the estimation result estimated in the sound source position estimating step.

  Further, a program according to the present invention is provided with a display unit capable of displaying an image, and a plurality of sets provided at different positions and collecting voices emitted from each of a plurality of sound sources to generate voice data. A sound source position estimation step of estimating the positions of the plurality of sound sources based on the voice data generated by each of the plurality of sound collection units in an information acquisition device including a sound unit; And a display control step of causing the display unit to display sound source position information related to the position of each of the plurality of sound sources, based on the estimation result estimated in (4).

  According to the present invention, it is possible to intuitively grasp the position of the speaker at the time of recording.

FIG. 1 is a diagram showing a schematic configuration of a transcriber system according to a first embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the transcriber system according to the first embodiment of the present invention. FIG. 3 is a flowchart showing an outline of processing performed by the information acquisition device according to Embodiment 1 of the present invention. FIG. 4 is a diagram showing a usage scene of the information acquisition device according to Embodiment 1 of the present invention. FIG. 5 is a bird's-eye view schematically showing the situation of FIG. FIG. 6 is a diagram schematically showing the position of the sound source estimated by the sound source position estimation unit according to Embodiment 1 of the present invention. FIG. 7 is a diagram schematically showing a calculation situation in which the sound source position estimation unit according to Embodiment 1 of the present invention calculates the arrival time difference for one sound source. FIG. 8 is a diagram schematically showing an example of a calculation method for calculating the arrival time difference calculated by the sound source position estimation unit according to Embodiment 1 of the present invention. FIG. 9 is a flowchart showing an outline of each sound source position display determination process of FIG. 3. FIG. 10 is a flowchart showing an outline of the icon determination generation process of FIG. FIG. 11 is a view schematically showing an example of an icon generated by the sound source information generation unit according to Embodiment 1 of the present invention. FIG. 12 is a diagram schematically showing another example of the icon generated by the sound source information generation unit according to Embodiment 1 of the present invention. FIG. 13 is a diagram schematically showing another example of the icon generated by the sound source information generation unit according to Embodiment 1 of the present invention. FIG. 14: is a figure which shows typically an example of the sound source positional information which the display control part which concerns on Embodiment 1 of this invention displays. FIG. 15 is a flowchart showing an outline of processing performed by the information processing apparatus according to Embodiment 1 of the present invention. FIG. 16 is a view schematically showing an example of a document creation screen according to Embodiment 1 of the present invention. FIG. 17 is a flowchart showing an outline of the keyword determination process of FIG. FIG. 18 is a view schematically showing another example of the document creation screen according to the first embodiment of the present invention. FIG. 19 is a view schematically showing another example of the document creation screen according to the first embodiment of the present invention. FIG. 20 is a view schematically showing another example of the document creation screen according to the first embodiment of the present invention. FIG. 21 is a perspective view showing a schematic configuration before an external microphone is attached to the information acquisition apparatus according to the second embodiment of the present invention. FIG. 22 is a perspective view showing a schematic configuration after an external microphone is attached to the information acquisition apparatus according to the second embodiment of the present invention. FIG. 23 is a diagram schematically showing the position of the sound source estimated by the sound source position estimation unit according to Embodiment 2 of the present invention. FIG. 24: is a figure which shows typically the calculation condition which the sound source position estimation part which concerns on Embodiment 2 of this invention calculates an arrival time difference with respect to one sound source.

  Hereinafter, modes for carrying out the present invention (hereinafter referred to as “embodiments”) will be described in detail with reference to the drawings. Note that the present invention is not limited by the following embodiments. In addition, the drawings referred to in the following description merely schematically show the shapes, sizes, and positional relationships to the extent that the contents of the present invention can be understood. That is, the present invention is not limited to only the shapes, sizes, and positional relationships illustrated in the respective drawings.

Embodiment 1
[Configuration of Transcriber System]
FIG. 1 is a diagram showing a schematic configuration of a transcriber system according to a first embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the transcriber system according to the first embodiment of the present invention.

  The transcriber system 1 shown in FIGS. 1 and 2 is, for example, a recording device such as a cellular phone or the like, which receives voice by using a microphone or the like and records voice data by using an IC recorder or a microphone or the like. The system includes an information acquisition device 2 that functions, and an information processing device 3 such as a personal computer that acquires voice data from the information acquisition device 2 via the communication cable 4 and performs transcription of voice data and various processes. In the first embodiment, the information acquisition device 2 and the information processing device 3 communicate bi-directionally via the communication cable 4. However, the present invention is not limited to this, and bi-directional communication can be performed wirelessly. It may be connected. In this case, the wireless communication standards include IEEE 802.11a, IEEE 802.11b, IEEE 802.11n, IEEE 802.11 g, IEEE 802.11 ac, Bluetooth (registered trademark), an infrared communication standard, and the like.

[Configuration of information acquisition device]
First, the configuration of the information acquisition device 2 will be described.
The information acquisition device 2 includes a first sound collection unit 20, a second sound collection unit 21, an external input detection unit 22, a display unit 23, a clock 24, an input unit 25, a recording unit 26, and a communication unit. 27, an output unit 28, and a device control unit 29.

  The first sound collecting unit 20 is provided on the left side of the top surface of the information acquisition device 2 (see FIG. 1). The first sound collection unit 20 picks up the sound emitted from each of the plurality of sound sources and converts it into an analog sound signal (electric signal), and performs A / D conversion processing and gain adjustment processing on this sound signal. To generate digital audio data (first audio data) and output it to the device control unit 29. The first sound collection unit 20 is configured using a microphone of any one of a unidirectional microphone, a nondirectional microphone and a bidirectional microphone, an A / D conversion circuit, a signal processing circuit, and the like.

  The second sound collecting unit 21 is provided at a position different from that of the first sound collecting unit 20. The second sound collecting unit 21 is provided on the right side of the top surface of the information acquisition device 2 separated from the first sound collecting unit 20 by a predetermined distance d (see FIG. 1). The second sound collection unit 21 picks up the sound emitted from each of the plurality of sound sources and converts it into an analog sound signal (electric signal), and performs A / D conversion processing and gain adjustment processing on this sound signal. To generate digital audio data (second audio data) and output it to the device control unit 29. The second sound collection unit 21 has a configuration similar to that of the first sound collection unit 20, and includes one of a unidirectional microphone, a nondirectional microphone and a bidirectional microphone, an A / D conversion circuit, and It is configured using a signal processing circuit or the like. In the first embodiment, the first sound collection unit 20 and the second sound collection unit 21 form a stereo microphone.

  The external input detection unit 22 inserts and removes the plug of the external microphone inserted from the outside of the information acquisition device 2, detects the insertion of the external microphone, and outputs the detection result to the device control unit 29. Further, the external input detection unit 22 receives an input of an analog audio signal (electric signal) generated by collecting an audio emitted from each of the plurality of sound sources by the external microphone, and the input is received as an audio signal. On the other hand, A / D conversion processing and gain adjustment processing are performed to generate digital audio data (including at least third audio data) and output it to the device control unit 29. Further, when the plug of the external microphone is inserted, the external input detection unit 22 outputs a signal indicating that the external microphone is connected to the information acquisition device 2 to the device control unit 29, and the voice generated by the external microphone is generated. The data is output to the device control unit 29. The external input detection unit 22 is configured using a microphone jack, an A / D conversion circuit, a signal processing circuit, and the like. Also, the external microphone is configured using any microphone such as a unidirectional microphone, a nondirectional microphone and a bidirectional microphone, and a stereo microphone capable of collecting left and right sounds. When a stereo microphone is used as the external microphone, the external input detection unit 22 generates two audio data (third audio data and fourth audio data) collected by each of the left and right microphones, and sends it to the device control unit 29. Output.

  The display unit 23 displays various information on the information acquisition device 2 under the control of the device control unit 29. The display unit 23 is configured using an organic EL (Electro Luminescence), a liquid crystal, or the like.

  The clock 24 generates time and date information on the date and time of the audio data generated in each of the first sound collecting unit 20 and the second sound collecting unit 21 and the external microphone, in addition to the clocking function. Output to

  The input unit 25 receives an input of various information related to the information acquisition device 2. The input unit 25 is configured using a button, a switch, and the like. Further, the input unit 25 is provided so as to be superimposed on the display area of the display unit 23, and has a touch panel 251 that detects a touch of an object from the outside and receives an input of an operation signal according to the detected position.

  The recording unit 26 is configured using a volatile memory, a non-volatile memory, a recording medium, and the like, and records an audio file storing audio data and various programs executed by the information acquisition device 2. The recording unit 26 includes a program recording unit 261 that records various programs executed by the information acquisition device 2 and an audio file recording unit 262 that records an audio file storing audio data. The recording unit 26 may be a recording medium such as a memory card that can be attached and detached from the outside.

  The communication unit 27 transmits data including an audio file in which audio data is stored to the information processing device 3 in accordance with a predetermined communication standard, and receives various information and data from the information processing device 3.

  The output unit 28 performs D / A conversion processing on digital audio data input from the device control unit 29, converts the digital audio data into an analog audio signal, and outputs the analog audio signal to the outside. The output unit 28 is configured using a speaker, a D / A conversion circuit, and the like.

  The device control unit 29 comprehensively controls the respective units constituting the information acquisition device 2. The device control unit 29 is configured using a central processing unit (CPU), a field programmable gate array (FPGA), or the like. The device control unit 29 includes a signal processing unit 291, a text conversion unit 292, a text conversion control unit 293, a voice determination unit 294, a sound source position estimation unit 295, a display position determination unit 296, and a voice print determination unit 297. A sound source information generation unit 298, a voice specification unit 299, a movement determination unit 300, an index addition unit 301, a voice file generation unit 302, and a display control unit 303.

  The signal processing unit 291 performs adjustment processing of the sound level of the sound data generated by the first sound collection unit 20 and the second sound collection unit 21, noise reduction processing, gain adjustment processing, and the like.

  The text conversion unit 292 generates voice text data composed of a plurality of characters by performing voice recognition processing on voice data. The details of the speech recognition process will be described later.

  When the textification control unit 293 receives an instruction signal for causing the textification unit 292 to generate voice text data from the input unit 25, the textification unit 292 performs speech only for a predetermined time after the input of the instruction signal is received. Generate text data.

  The voice determination unit 294 determines whether or not the silent period is included in the voice data sequentially subjected to the automatic level adjustment by the signal processing unit 291. Specifically, the voice determination unit 294 determines whether the voice level of the voice data is less than a predetermined threshold, and determines that a period less than the predetermined threshold is a silent period.

  The sound source position estimation unit 295 estimates the positions of the plurality of sound sources based on the audio data generated by each of the first sound collection unit 20 and the second sound collection unit 21. Specifically, based on the audio data generated by each of the first sound collecting unit 20 and the second sound collecting unit 21, the sound signal emitted from each of the plurality of sound sources is transmitted to the first sound collecting unit 20 and the second sound collecting unit 20. The difference in arrival time to reach each of the sound collection units 21 is calculated, and the position of each of the plurality of sound sources when the information acquisition device 2 is viewed from the center is estimated based on the calculation result.

  The display position determination unit 296 determines the display position of each of the plurality of sound sources in the display area of the display unit 23 based on the shape of the display area of the display unit 23 and the estimation result estimated by the sound source position estimation unit 295. Specifically, the display position determination unit 296 determines the display position of each of the plurality of sound sources when the center of the display area of the display unit 23 is the information acquisition device 2. For example, the display position determination unit 296 divides each display area of the display unit 23 into four quadrants, and displays each of a plurality of sound sources when the information acquisition device 2 is placed at the center of the display area of the display unit 23 Determine the position.

  The voiceprint determination unit 297 determines the voiceprint of each of the plurality of sound sources based on the voice data. Specifically, the voiceprint determination unit 297 determines each voiceprint (speaker) of the plurality of sound sources included in the voice data. For example, before the voice print determination unit 297 performs recording of a conference using the information acquisition device 2, the voice print determination unit 297 generates voice data based on a speaker identification template in which features based on voice emitted by speakers participating in the conference are registered. A voiceprint (speaker) is determined from each of a plurality of included sound sources. Further, the voiceprint determination unit 297 determines, based on voice data, frequency height (voice height), intonation, volume (intensity), histogram, etc. in addition to the feature based on the voice emitted by the speaker. . Of course, the voiceprint determination unit 297 may determine gender based on voice data. Furthermore, the voiceprint determination unit 297 may determine the volume (intensity) and the frequency height (voice height) of each speaker emitted by each of the plurality of speakers based on the voice data. Of course, the voiceprint determination unit 297 may determine the intonation for each speaker generated by each of the plurality of speakers based on the voice data.

  The sound source information generation unit 298 generates a plurality of sound source information on each of the plurality of sound sources based on the determination result determined by the voiceprint determination unit 297. Specifically, the sound source information generation unit 298 generates voice information for each speaker emitted by the speaker determined by the voiceprint determination unit 297. For example, the sound source information generation unit 298 generates, as voice information, an icon schematically showing the speaker based on the height of the frequency (voice height) emitted by the speaker. Note that the sound source information generation unit 298 may change and generate the type of voice information, for example, a female icon, a male icon, or an icon such as a dog or a cat based on the gender determined by the voiceprint determination unit 297. Here, the sound source information generation unit 298 may prepare voice data of a specific height as a database, and compare this with the acquired voice signal to determine an icon, and a plurality of detected speakers The comparison may be performed using, for example, the frequency height (voice height) of each of the plurality of voices. In addition, the sound source information generation unit 298 may create a database of types and phrases used, such as gender, language, age, etc., and compare this with a voice pattern to determine an icon. In addition, there is also a problem as to whether or not even the icon of a person who just just hit a match is shown by just saying something irrelevant. Since such remarks are often less likely to be heard later and are like supplements to the main remarks, the sound source information generation unit 298 has little meaning in generating icons. Intuitive judgment may be rather inappropriate. Therefore, the sound source information generation unit 298 does not consider it as an important utterance and considers it as an ambiguous utterance if the utterance is not longer than a specific length of time, or if the subject, object nouns, nouns and adjectives are not clear. The visibility may be made different by not making the speaker iconify, thinning the icon, making it dotted line or smaller, or making the lines constituting the icon break in the middle. That is, the sound source information generation unit 298 has a function of judging the content of the speech by speech recognition, judging the used words, and grammatically verifying the degree of completion of the speech. It may be possible to determine whether it is the subject. As to whether or not the words related to such an agenda, it is detected whether words similar to the content of the utterance of the person (the chairperson or facilitator) who is mainly speaking are used, etc. You may compare and determine. If this does not match, it may be an unclear speech, and it can be similarly judged whether the voice is small or the speech is short. With such a device, the visibility of the icon schematically showing the speaker is generated based on the sound source information of each speaker uttered by the speaker based on the length and clarity of the voice uttered by the speaker Improve intuitive searchability of speech. Also, even if the sound source information generation unit 298 generates an icon schematically showing each of the plurality of speakers as sound source information based on the comparison of the volume of each of the plurality of speakers determined by the voiceprint determination unit 297. Good. Of course, based on the voice data, the sound source information generation unit 298 makes the icons schematically showing the speakers different from each other, based on the length and volume of the voice for each speaker emitted by each of the plurality of speakers. Sound source information may be generated.

  The voice identification unit 299 identifies an appearance position (appearance time) at which each of the plurality of voiceprints determined by the voiceprint determination unit 297 appears on voice data.

  The movement determination unit 300 determines whether each of the plurality of sound sources is moving, based on the estimation result estimated by the sound source position estimation unit 295 and the determination result determined by the voiceprint determination unit 297.

  The index adding unit 301 adds, to the silent period determined by the voice determination unit 294, an index for distinguishing from the other periods to the voice data to at least one of the first and the last of the silent period.

  The voice file generation unit 302 includes voice data subjected to signal processing by the signal processing unit 291, sound source position information estimated by the sound source position estimation unit 295, a plurality of sound source information generated by the sound source information generation unit 298, and voice specification. Position information on the appearance position specified by the unit 299, position information on the position of the index added by the index adding unit 301, or time information on time on the voice data to which the index is added, and voice text data generated by the textification unit 292 An associated audio file is generated and recorded in the audio file recording unit 262. In addition, the audio file generation unit 302 causes the textification unit 292 to generate audio text data for a predetermined time from the time when the input unit 25 receives the audio data on which the signal processing unit 291 has performed the signal processing and the input unit 25 receives the instruction signal. An audio file in which candidate timing information as candidate timing is associated may be generated and recorded in the audio file recording unit 262 functioning as a recording medium.

  The display control unit 303 controls the display mode of the display unit 23. Specifically, the display control unit 303 causes the display unit 23 to display various information related to the information acquisition device 2. For example, the display unit 23 causes the display unit 23 to display the audio level of the audio data adjusted by the signal processing unit 291. In addition, the display control unit 303 causes the display unit 23 to display sound source position information on each position of a plurality of sound sources based on the estimation result estimated by the sound source position estimating unit 295. Specifically, the display control unit 303 causes the display unit 23 to display the sound source position information based on the determination result determined by the display position determination unit 296. More specifically, the display control unit 303 causes the display unit 23 to display a plurality of pieces of sound source information generated by the sound source information generation unit 298 as sound source position information.

[Configuration of Information Processing Apparatus]
Next, the configuration of the information processing device 3 will be described.
The information processing device 3 includes a communication unit 31, an input unit 32, a recording unit 33, an audio reproduction unit 34, a display unit 35, and an information processing control unit 36.

  The communication unit 31 transmits data to the information acquisition device 2 in accordance with a predetermined communication standard, and receives from the information acquisition device 2 data including at least an audio file in which audio data is stored.

  The input unit 32 receives an input of various information related to the information processing device 3. The input unit 32 is configured using a button, a switch, a keyboard, a touch panel, and the like. For example, when the user performs a documentization operation, the input unit 32 receives an input of text data.

  The recording unit 33 is configured using a volatile memory, a non-volatile memory, a recording medium, and the like, and records an audio file storing audio data and various programs executed by the information processing apparatus 3. The recording unit 33 has a program recording unit 331 for recording various programs to be executed by the information processing device 3 and a voice-to-text dictionary data recording unit 332 used to convert voice data into text data. It is preferable that this is a database that can search not only the relationship between speech and text but also synonyms and the like. Here, synonyms are two or more words whose word forms are different but their meanings are similar to each other in the same language, and in some cases, substitution is possible. Of course, thesaurus and synonyms may be included.

  The audio reproduction unit 34 performs D / A conversion processing on digital audio data input from the information processing control unit 36, converts it into an analog audio signal, and outputs the signal to the outside. The audio reproduction unit 34 is configured using a speaker, a D / A conversion circuit, and the like.

  Under the control of the information processing control unit 36, the display unit 35 displays a time bar corresponding to various information related to the information processing device 3 and a recording time of audio data. The display unit 35 is configured using an organic EL, a liquid crystal, or the like.

  The information processing control unit 36 centrally controls the units that constitute the information processing apparatus 3. The information processing control unit 36 is configured using a CPU or the like. The information processing control unit 36 includes a text conversion unit 361, a specification unit 362, a keyword determination unit 363, a keyword setting unit 364, a voice control unit 365, a display control unit 366, and a document generation unit 367. .

  The text conversion unit 361 generates voice text data composed of a plurality of characters by performing voice recognition processing on voice data. The details of the speech recognition process will be described later.

  The specifying unit 362 specifies an appearance position (appearing time) on audio data in which the character string of the keyword matches the character string in the audio text data. Of course, the specification unit 362 does not have to completely match the character string of the keyword and the character string in the audio text data, for example, the similarity (for example, 80% or more) between the character string of the keyword and the character string in the audio text data The appearance position (appearance time) on high voice data may be specified.

  The keyword determination unit 363 determines whether the voice file acquired by the communication unit 31 from the information acquisition device 2 includes a keyword candidate. Specifically, the keyword determination unit 363 determines whether or not the audio text data is stored in the audio file acquired by the communication unit 31 from the information acquisition device 2.

  When the keyword setting unit 364 determines that the voice file acquired from the information acquisition device 2 via the communication unit 31 is a keyword candidate, the keyword setting unit 364 causes the keyword candidate stored in the voice file to appear on the voice data Set as a keyword to search for the position you want. Specifically, the keyword setting unit 364 sets voice text data stored in the voice file acquired by the communication unit 31 from the information acquisition device 2 as a keyword for searching for a position where it appears on the voice data. When the meeting is over, even though the image of the word is remembered, the correct word itself is often forgotten. Therefore, the keyword setting unit 364 is a synonym (for example, if the word is "important", A similar word such as “point” or “important” may be searched by dictionary search in a database (speech / textification dictionary data recording unit 332) or the like to search for a keyword having a similar meaning.

  The voice control unit 365 controls the drive of the voice reproduction unit 34. Specifically, the audio control unit 365 causes the audio reproduction unit 34 to reproduce the audio data stored in the audio file.

  The display control unit 366 controls the display mode of the display unit 35. The display control unit 366 causes the display unit 35 to display position information regarding the appearance position where the keyword appears on the time bar.

[Process of information acquisition device]
Next, the process performed by the information acquisition device 2 will be described. FIG. 3 is a flowchart showing an outline of the process performed by the information acquisition device 2. FIG. 4 is a view showing a usage scene of the information acquisition device 2. FIG. 5 is a bird's-eye view schematically showing the situation of FIG.

  As shown in FIG. 3, when the instruction signal instructing recording is input from the input unit 25 by operating the input unit 25 (step S101: Yes), the device control unit 29 selects the first sound collection unit 20. The second sound pickup unit 21 is driven to sequentially store the voice data in the voice file according to the voice input and start the recording to be recorded in the recording unit 26 (step S102).

  Subsequently, the signal processing unit 291 performs automatic level adjustment to automatically adjust the level of the audio data generated by each of the first sound collecting unit 20 and the second sound collecting unit 21 (step S103).

  After that, the display control unit 303 causes the display unit 23 to display the level of the automatic level adjustment performed on the audio data by the signal processing unit 291 (step S104).

  Subsequently, the voice determination unit 294 determines whether or not the silent period is included in the voice data sequentially subjected to the automatic level adjustment by the signal processing unit 291 (step S105). Specifically, the sound determination unit 294 determines whether the volume level is less than a predetermined threshold value for each predetermined frame period of the sound data in which the signal processing unit 291 has performed the automatic level adjustment sequentially. It is determined whether a silent period is included. More specifically, when there is a predetermined period (for example, 10 seconds) in which the sound volume level of the sound data is less than the predetermined threshold, the sound determination unit 294 determines that the sound data includes a silent period. The user can appropriately set the predetermined period using the input unit 25. When it is determined by the voice determination unit 294 that the voice data to which the signal processing unit 291 has sequentially performed the automatic level adjustment includes a silent period (step S105: Yes), the information acquisition device 2 proceeds to step S106 described later. Transition. On the other hand, when it is determined by the voice determination unit 294 that the voice data for which the signal processing unit 291 has sequentially performed the automatic level adjustment does not include the silent period (step S105: No), the information acquisition device 2 It transfers to step S107 mentioned later.

  In step S106, the index adding unit 301 adds, to the voice data, in the silent period determined by the voice determining unit 294, an index for distinguishing it from other periods to at least one of the beginning and the end of the silent period. . After step S106, the information acquisition device 2 proceeds to step S107 described later.

  In step S107, when an instruction signal instructing setting of a keyword candidate for adding an index is input from the input unit 25 by operating the input unit 25 (step S107: Yes), the information acquisition device 2 It transfers to step S108 mentioned later. On the other hand, when an instruction signal instructing setting of a keyword candidate for adding an index is not input from the input unit 25 (step S107: No), the information acquisition device 2 proceeds to step S109 described later. . This step corresponds to the case where the user gives some instruction such as a note or a sticky note when an important topic that he / she wants to hear back later has begun while recording, such as during a meeting. . Here, a description such as a specific switch operation (for example, an input by an operation to the input unit 25) is described, but a similar input may be performed when a voice such as "here important" is detected by voice. . That is, the index adding unit 301 generates an index based on the text data converted into text by the text converting unit 292 with respect to the voice data input through the first sound collecting unit 20 and the second sound collecting unit 21. It may be added.

  At such a timing, since there is a high possibility that discussions using words that are important keywords in the meeting have started thereafter, the textification control unit 293 instructs the setting of keyword candidates from the input unit 25 in step S108. Voice recognition processing to be described later in the textification unit 292 for voice data retrospectively for a predetermined time (for example, 3 seconds, if the conversation is not interrupted, processing to go back after that) may be performed from the time when the instruction signal is input To perform textification and generate speech text data. This makes it possible to make it easy to determine in a recording site the keywords that you want to rehearse later in real time. When the meeting is over, even though you remember the image of a word, you often forget exactly the word itself. In this way, when searching later, it is easy to understand the timing to look for often. This can be said to be a candidate timing, so it is highly probable that at this timing there is discussion using important keywords, their synonyms, and similar nuance words. For this reason, the textification control unit 293 executes textification to generate speech text data because it is better for the speech data here to be visualized as text preferentially if it can be understood as a whole of the discussion. Let In step S108, the index adding unit 301 does not necessarily perform processing until the conversion to text, and records the timing for searching intensively, the timing of minutes and seconds from the start of recording, and the candidate timing in association with the audio data. You may do just that. There is a method of recording candidate timing information as metadata at the time of creating an audio file. After step S108, the information acquisition device 2 proceeds to step S109 described later.

  In step S109, the sound source position estimating unit 295 estimates the positions of the plurality of sound sources based on the audio data generated by each of the first sound collecting unit 20 and the second sound collecting unit 21. After step S109, the information acquisition device 2 proceeds to step S110 described later.

  FIG. 6 is a diagram schematically showing the position of the sound source estimated by the sound source position estimation unit 295. As shown in FIG. As shown in FIG. 6, the sound source position estimation unit 295 is configured to generate a speaker that is the sound source A1 and a speaker that is the sound source A2 based on the sound data generated by each of the first sound collection unit 20 and the second sound collection unit 21. The sound source direction is calculated by calculating the arrival time difference at which the voice emitted by each person reaches each of the first sound collection unit 20 and the second sound collection unit 21 and forming the focus of the sound source using the calculated arrival time difference. presume.

  FIG. 7 is a diagram schematically showing a calculation situation in which the sound source position estimation unit 295 calculates the arrival time difference for one sound source. FIG. 8 is a diagram schematically showing an example of a calculation method for calculating the arrival time difference calculated by the sound source position estimation unit 295. As shown in FIG.

As shown in FIGS. 7 and 8, the sound source position estimation unit 295 sets the distance between the first sound collection unit 20 and the second sound collection unit 21 to d, the sound source direction of the speaker who is the sound source A1, θ, and the sound speed. In the case of V, the arrival time difference T is calculated by the following equation (1).
T = (d × COS (θ)) / V (1)
In this case, the sound source position estimation unit 295 calculates the arrival time difference T using the degree of coincidence of the frequencies included in the two audio data generated by each of the first sound collection unit 20 and the second sound collection unit 21. Can. Therefore, the sound source position estimation unit 295 calculates the arrival time difference T using the degree of coincidence of the frequencies included in the two audio data generated by each of the first sound collection unit 20 and the second sound collection unit 21. After that, the sound source position estimation unit 295 estimates the direction of the sound source by calculating the sound source direction θ using the arrival time difference T and the equation (1). Specifically, the sound source position estimation unit 295 estimates the direction of the sound source A1 by calculating the sound source direction θ by the following equation (2).
θ = COS ⁻¹ (T × V) / d (2)
As described above, the sound source position estimation unit 295 can estimate the direction for each sound source.

Returning to FIG. 3, the description of step S110 and subsequent steps is continued.
In step S110, the information acquisition device 2 determines the position for displaying the sound source position information on each of the positions of the plurality of sound sources on the display area of the display unit 23, based on the estimation result of the sound source position estimation unit 295. Each sound source position display determination process is executed.

[Each sound source position display decision processing]
FIG. 9 is a flowchart showing an outline of each sound source position display determination process of step S110 of FIG.

  As shown in FIG. 9, the voiceprint determination unit 297 determines the type of each of the plurality of sound sources based on the voice data (step S201). Specifically, the voiceprint determination unit 297 analyzes the voice emitted by each of the plurality of sound sources estimated by the sound source position estimation unit 295 using known voiceprint authentication technology from voice data, and separates the voice source Determine the type of each. For example, the voiceprint determination unit 297 determines a voiceprint (speaker) from each of the plurality of sound sources included in the voice data, based on the speaker identification template in which the feature based on the voice emitted by the speaker participating in the conference is registered. Do.

  Based on the shape of the display area of the display unit 23 and the positions of the plurality of sound sources estimated by the sound source position estimation unit 295, the display position determination unit 296 determines that each sound source on the display area of the display unit 23 starts from the first quadrant. It is determined whether it is located in any of the fourth quadrant (step S202). Specifically, the display position determination unit 296 determines the display position of each sound source when the center of the display area of the display unit 23 is viewed as the information acquisition device 2. For example, the display position determination unit 296 determines whether each of the plurality of sound sources estimated by the sound source position estimation unit 295 is located in any one of the first quadrant to the fourth quadrant. In this case, the display position determination unit 296 is a two straight line passing through the center of the display area of the display unit 23 with respect to the display area of the display unit 23 and divided by two straight lines orthogonal to each other on a plane. Divide the first quadrant into the fourth quadrant. In the present embodiment, the display position determination unit 296 is divided into four quadrants. However, the display position determination unit 296 is not limited to this and may be two quadrants and provided in the information acquisition device 2 It may be made to be able to select suitably according to the number of sound collection parts.

  Subsequently, the display position determination unit 296 determines whether or not there are a plurality of sound sources in the same quadrant (step S203). When the display position determination unit 296 determines that there are a plurality of sound sources in the same quadrant (step S203: Yes), the information acquisition device 2 proceeds to step S204 described later. On the other hand, when it is determined by the display position determination unit 296 that there are not a plurality of sound sources in the same quadrant (step S203: No), the information acquisition device 2 proceeds to step S205 described later.

  In step S204, the display position determination unit 296 determines whether or not each sound source located in the same quadrant has a perspective. If it is determined by the display position determination unit 296 that the sound sources located in the same quadrant have a distance (step S204: Yes), the information acquisition device 2 proceeds to step S206 described later. On the other hand, when it is determined by the display position determination unit 296 that the respective sound sources located in the same quadrant do not have perspective (step S204: No), the information acquisition device 2 proceeds to step S205 described later.

  In step S205, the display position determination unit 296 determines the display position at which the icon is displayed, based on the sound source of each quadrant. After step S205, the information acquisition device 2 proceeds to step S207 described later.

  In step S206, the display position determination unit 296 determines the display position at which the icon is displayed, based on the perspective of each of the plurality of sound sources located in the same quadrant. After step S206, the information acquisition device 2 proceeds to step S207 described later.

[Icon determination generation process]
FIG. 10 is a flowchart showing an outline of the icon determination generation process of step S207 of FIG.

  As shown in FIG. 10, first, the sound source information generation unit 298 determines the order in descending order of voice among a plurality of voiceprints determined by the voiceprint determination unit 297 (step S301).

  Subsequently, the sound source information generation unit 298 generates a speaker (sound source) of the highest voice among the plurality of voiceprints determined by the voiceprint determination unit 297 as a thin face long hair icon (step S302). Specifically, as shown in FIG. 11, the sound source information generation unit 298 sets the speaker (sound source) of the highest voice among the plurality of voiceprints determined by the voiceprint determination unit 297 to a thin-faced long hair icon O1. Generate as (an icon that takes a woman as an image).

  Thereafter, the sound source information generation unit 298 generates a speaker (sound source) of the lowest voice among the plurality of voiceprints determined by the voiceprint determination unit 297 as a round face short hair icon (step S303). Specifically, as shown in FIG. 12, the sound source information generation unit 298 sets the speaker (sound source) of the lowest voice of the plurality of voiceprints determined by the voiceprint determination unit 297 to a round-haired short hair icon O 2. Generate as (an icon that takes men as an image).

  Subsequently, the sound source information generation unit 298 generates icons in the order of the heights of the plurality of voiceprints determined by the voiceprint determination unit 297 (step S304). Specifically, the sound source information generation unit 298 sequentially deforms the shape of the face from a thin face to a round face in the order of heights of a plurality of voiceprints determined by the voiceprint determination unit 297 and also turns from long hair to short hair. Generate an icon that has been transformed sequentially. Here, although a business scene is assumed, it may be a child to have a meeting, so the sound source information generation unit 298 uses a different icon generation method if it is a feature of the child's voice. For example, the sound source information generation unit 298 determines that such a situation is present based on differences in voice quality when being with an adult, and makes the child a smaller icon or has more children. You may perform application which makes identification high, such as expressing an adult large. Since the child is in the process of growth, the aspect ratio of the face is generally closer to 1: 1 than that of the adult, so it may be possible to emphasize the width of the icon and make it wider. That is, at the time of icon creation, the sound source information generation unit 298 may generate an icon in which the horizontal width is emphasized.

  After that, the movement determination unit 300 determines a plurality of voiceprints determined by the voiceprint determination unit 297 based on the positions of the plurality of sound sources estimated by the sound source position estimation unit 295 and the plurality of voiceprints determined by the voiceprint determination unit 297. It is determined whether there is a moving sound source moving in two or more quadrants of the first quadrant to the fourth quadrant in the voiceprint (step S305). Specifically, the movement determination unit 300 determines whether the position of the sound source of each quadrant determined by the display position determination unit 296 and the position of each sound source estimated by the sound source position estimation unit 295 differ with the passage of time, If they differ with the passage of time, it is determined that there is a moving sound source. If it is determined by the movement determination unit 300 that there is a sound source moving in each quadrant (step S305: Yes), the information acquisition device 2 proceeds to step S306 described later. On the other hand, when it is determined by the movement determination unit 300 that there is no sound source moving in each quadrant (step S305: No), the information acquisition device 2 returns to the subroutine of FIG. 9 described above.

  In step S306, the voice identification unit 299 identifies an icon corresponding to the sound source determined by the movement determination unit 300. Specifically, the voice identification unit 299 identifies an icon of a sound source moving in two or more quadrants of the first quadrant to the fourth quadrant determined by the movement determination unit 300.

  Subsequently, the sound source information generation unit 298 adds movement information to the icon of the sound source specified by the sound specification unit 299 (step S307). Specifically, as shown in FIG. 13, the sound source information generation unit 298 adds the movement icon U1 (movement information) to the icon O2 of the sound source specified by the sound specification unit 299. Although the sound source information generation unit 298 adds the move icon U1 to the moved icon O2, for example, the color of the icon O2 may be changed or the shape may be changed. Of course, the sound source information generation unit 298 may add a character or a figure to the moved icon O2, or may add a moving time, a moved timing, or the like. After step S307, the information acquisition device 2 returns to the above-described subroutine of FIG.

Returning to FIG. 9, the description of step S208 and subsequent steps is continued.
In step S208, when the determination of all the quadrants is completed (step S208: Yes), the information acquisition device 2 returns to the main routine of FIG. On the other hand, when the determination of all the quadrants is not completed (step S208: No), the information acquisition device 2 returns to the above-described step S203.

Returning to FIG. 3, the description of step S111 and subsequent steps is continued.
In step S111, the display control unit 303 causes the display unit 23 to display the plurality of sound source position information generated in step S110 described above. Specifically, as shown in FIG. 14, the display control unit 303 displays the display area of the display unit 23 by superimposing each of the icons O1 to O3 on each of the first quadrant H1 to the fourth quadrant H4. It is displayed on the part 23. Thereby, even at the time of recording, the user can intuitively grasp the position of the speaker (sound source) when viewed from the information acquisition device 2 at the center. Furthermore, since the move icon U1 is superimposed on the icon O2, the user can intuitively grasp the speaker who has moved at the time of recording.

  In step S112, when an instruction signal to end recording is input from the input unit 25 (step S112: Yes), the information acquisition device 2 proceeds to step S113 described later. On the other hand, when the instruction | indication signal which complete | finishes recording from the input part 25 is not input (step S112: No), the information acquisition apparatus 2 returns to step S103 mentioned above.

  In step S113, the voice data subjected to signal processing by the signal processing unit 291, the sound source position information estimated by the sound source position estimating unit 295, the plurality of sound source information generated by the sound source information generating unit 298, and the sound specifying unit 299 The specified appearance position, the position information on the position of the index added by the index adding unit 301, or the time information on time on the voice data to which the index is added are associated with the voice text data generated by the textification unit 292 An audio file is generated and recorded in the audio file recording unit 262. After step S113, the information acquisition device 2 proceeds to step S114 described later. Note that the audio file generation unit 302 causes the textification unit 292 to generate audio text data for a predetermined time from the time when the input unit 25 receives the audio data on which the signal processing unit 291 has performed the signal processing and the input unit 25 receives the instruction signal. An audio file in which candidate timing information as candidate timing is associated may be generated and recorded in the audio file recording unit 262. That is, the audio file generation unit 302 generates an audio file by correlating the audio data with candidate timing information which is candidate timing for a predetermined period of time from the time when the input unit 25 receives the input of the instruction signal. You may make it record on the recording part 262.

  Subsequently, when the instruction signal to turn off the power is input from the input unit 25 (step S114: Yes), the information acquisition device 2 ends the process. On the other hand, when the instruction | indication signal which turns off a power supply is not input from the input part 25 (step S114: No), the information acquisition apparatus 2 returns to step S101 mentioned above.

  In step S101, when the instruction signal instructing recording is not input from the input unit 25 (step S101: No), the information acquisition device 2 proceeds to step S115.

  Subsequently, when an instruction signal instructing reproduction of the audio file is input from the input unit 25 (step S115: Yes), the information acquisition device 2 proceeds to step S116 described later. On the other hand, when the instruction signal instructing reproduction of the audio file is not input from the input unit 25 (step S115: No), the information acquisition device 2 proceeds to step S122.

  In step S116, when the audio file is selected by operating the input unit 25 (step S116: Yes), the information acquisition device 2 proceeds to step S117 described later. On the other hand, when the input unit 25 is not operated and the audio file is not selected (step S116: No), the information acquisition device 2 proceeds to step S114.

  In step S117, the display control unit 303 causes the display unit 23 to display the plurality of sound source position information stored in the audio file selected via the input unit 25.

  Subsequently, when any icon of the plurality of pieces of audio position information displayed by the display unit 23 is touched via the touch panel 251 (step S118: Yes), the output unit 28 reproduces audio data corresponding to the icon. And output (step S119).

  Thereafter, when an instruction signal for ending the reproduction of the audio file is input from the input unit 25 (step S120: Yes), the information acquisition device 2 proceeds to step S114. On the other hand, when the instruction signal for ending the reproduction of the audio file is not input from the input unit 25 (step S120: No), the information acquisition device 2 returns to the above-described step S117.

  In step S118, when any icon of the plurality of audio position information displayed by display unit 23 is not touched via touch panel 251 (step S118: No), output unit 28 reproduces audio data (step S118). S121). After step S121, the information acquisition device 2 proceeds to step S120.

  In step S122, when the instruction signal for transmitting the audio file is input by operating the input unit 25 (step S122: Yes), the communication unit 27 transmits the audio file to the information processing device 3 according to a predetermined communication standard. It transmits (step S123). After step S123, the information acquisition device 2 proceeds to step S114.

  In step S122, when the instruction signal for transmitting the audio file is not input by operating the input unit 25 (step S122: No), the information acquisition device 2 proceeds to step S114.

[Process of Information Processing Device]
Next, processing executed by the information processing device 3 will be described. FIG. 15 is a flowchart showing an outline of the process performed by the information processing device 3.

  As shown in FIG. 15, first, when the user performs a documentization operation for making a pick while reproducing audio data (step S401: Yes), the communication unit 31 acquires the information acquisition device connected to the information processing device 3. An audio file is acquired from 2 (step S402).

  Subsequently, the display control unit 366 causes the display unit 35 to display a document creation screen (step S403). Specifically, as shown in FIG. 16, the display control unit 366 causes the display unit 35 to display the document creation screen W1. The document creation screen W1 includes a display area R1, a display area R2, and a display area R3. In the display area R1, the text corresponding to the text data transcribed from the reproduction of the audio data by the user operating the input unit 32 is displayed. The display area R2 shows a time bar T1 corresponding to audio data stored in the audio file, a display area K1 for displaying a keyword input according to the operation of the input unit 32, and sound source information on a sound source at the time of recording. And a plurality of icons O1 to O3. The display area R3 has a time bar T2 corresponding to the audio data stored in the audio file, and a display area K2 for displaying the appearance position of the keyword.

  After that, when a reproduction operation to reproduce audio data is performed via the input unit 32 (step S404: Yes), the audio control unit 365 causes the audio reproduction unit 34 to reproduce the audio data stored in the audio file ( Step S405).

  Subsequently, the keyword determination unit 363 determines whether there is a keyword candidate in the audio file (step S406). Specifically, the keyword determination unit 363 determines whether one or more pieces of speech text data as a keyword are stored in the speech file. When it is determined by the keyword determination unit 363 that there is a keyword candidate in the audio file (step S406: Yes), the keyword setting unit 364 searches the position where the keyword candidate stored in the audio file appears on the audio data The keywords of are set (step S407). Specifically, the keyword setting unit 364 sets one or more voice text data stored in the voice file as a keyword for searching for a position where it appears on the voice data. After step S407, the information processing device 3 proceeds to step S410 described later. On the other hand, when the keyword determination unit 363 determines that there is no keyword candidate in the audio file (step S406: No), the information processing device 3 proceeds to step S408 described later. When the meeting is over, even though it remembers the image of the word that is the keyword but often forgets the exact word itself, the keyword determination unit 363 uses a dictionary or the like that records words having similar meanings. You may search for synonyms.

  When the specific keyword appearing in the audio data is searched through the input unit 32 (step S409: Yes), the information processing apparatus 3 determines whether the input unit 32 is operated (step S408). It transfers to step S410 mentioned later. On the other hand, when the input unit 32 is operated (step S408: Yes), when the specific keyword appearing in the audio data is not searched via the input unit 32 (step S409: No), the information processing device 3 The process proceeds to step S416 described later.

  In step S408, when the input unit 32 is not operated (step S408: No), the information processing device 3 proceeds to step S414 described later.

  In step S410, the information processing control unit 36 executes keyword determination processing to determine the time for which the keyword appears on the voice data.

[Keyword determination processing]
FIG. 17 is a flowchart showing an outline of the keyword determination process in step S410 of FIG. 15 described above.

  As shown in FIG. 17, when the automatic mode for automatically detecting a specific keyword appearing in voice data is set (step S501: Yes), the information processing device 3 proceeds to step S502 described later. On the other hand, when the automatic mode for automatically detecting the specific keyword appearing in the voice data is not set (step S501: No), the information processing device 3 proceeds to step S513 described later.

  In step S502, the text conversion unit 361 decomposes the speech data into speech waveforms (step S502), and generates speech text data by performing Fourier transform on the decomposed speech waveforms (step S503).

  Subsequently, the keyword determination unit 363 matches the voice text data subjected to the Fourier transformation by the textification unit 361 with any one of a plurality of phonemes included in the phoneme dictionary data recorded by the speech textification dictionary data recording unit 332 It is determined whether or not to do (step S504). Specifically, the keyword determination unit 363 selects one of a plurality of phonemes included in the phoneme dictionary data recorded by the speech-to-text dictionary data recording unit 332, as a result of the Fourier transformation performed by the textification unit 361. It is determined whether or not it matches the waveform. However, the keyword determination unit 363 does not have to be a strict match because there is a habit or difference in pronunciation depending on individuals, and it may be determined whether the degree of similarity is high. Also, some individuals may say the same thing differently, so you may search for synonyms as needed. The result of the Fourier transformation performed by the text conversion unit 361 by the keyword determination unit 363 matches one of a plurality of phonemes included in the phoneme dictionary data recorded by the speech text conversion dictionary data recording unit 332 (the similarity is high When it is determined that the information processing apparatus 3 is determined (step S504: Yes), the information processing device 3 proceeds to step S506 described later. On the other hand, the result that the text conversion unit 361 performs the Fourier transform by the keyword determination unit 363 does not match any one of a plurality of phonemes included in the phoneme dictionary data recorded by the voice text conversion dictionary data recording unit 332 When it is determined that the similarity is low (step S504: No), the information processing device 3 proceeds to step S505 described later.

  In step S505, the text conversion unit 361 changes the waveform width for performing Fourier transform on the decomposed speech waveform. After step S505, the information processing device 3 returns to step S503.

  In step S506, the text conversion unit 361 performs phonemicization as the result of the Fourier transform on the phonemes determined to be coincident by the keyword determination unit 363.

  Subsequently, the text conversion unit 361 creates a phoneme set including a plurality of phonemes (step S507).

  Thereafter, the keyword determination unit 363 determines whether the phoneme set created by the textification unit 361 matches any one of a plurality of words included in the speech-to-text dictionary data recorded by the speech-to-text dictionary data recording unit 332. (Whether or not the degree of similarity is high) is determined (step S508). The phoneme set created by the text conversion unit 361 by the keyword determination unit 363 matches one of a plurality of words included in the audio text conversion dictionary data recorded by the audio text conversion dictionary data storage unit 332 (the similarity is high (Step S508: Yes), the information processing apparatus 3 proceeds to step S510 described later. On the other hand, the phoneme set created by the text conversion unit 361 by the keyword determination unit 363 does not match any of a plurality of words included in the audio text conversion dictionary data recorded by the audio text conversion dictionary data recording unit 332 When it is determined that the degree of similarity is low (step S508: No), the information processing device 3 proceeds to step S509 described later.

  In step S509, the text conversion unit 361 changes the phoneme set configured by a plurality of phonemes. For example, the textification unit 361 changes the phoneme set by decreasing or increasing the number of phonemes. After step S509, the information processing device 3 returns to step S508 described above. An example of the process including the processes of step S502 to step S509 corresponds to the voice recognition process described above.

  In step S510, the identification unit 362 determines whether the character string of the keyword input through the input unit 32 matches the character string of the audio text data generated by the textification unit 361 (whether the similarity is high or not). To determine In this case, the identifying unit 362 determines whether the character string of the keyword set by the keyword setting unit 364 matches the character string of the audio text data generated by the text converting unit 361 (whether the similarity is high) May be determined. When the identification unit 362 determines that the character string of the keyword input through the input unit 32 matches the character string of the voice text data generated by the textification unit 361 (the similarity is high) (Step S510: Yes And the information processing apparatus 3 proceeds to step S511 described later. On the other hand, when the specification unit 362 determines that the character string of the keyword input through the input unit 32 does not match the character string of the voice text data generated by the textification unit 361 (the similarity is low). (Step S510: No), the information processing device 3 proceeds to Step S512 described later.

  In step S511, the identifying unit 362 identifies the appearance time of the keyword on the audio data. Specifically, the specifying unit 362 voices a period during which the character string of the keyword input through the input unit 32 and the character string of the voice text data generated by the textifying unit 361 match (high similarity) It specifies as the appearance position (appearance time) of the keyword on the data. However, the identifying unit 362 does not have to be an exact match because the pronunciation may or may not differ depending on the individual, and it may be determined whether the similarity is high. Further, depending on the individual, the same thing may be expressed differently, so the identifying unit 362 may perform a search using synonyms as necessary. This makes it possible to make it easy to determine in real time the keywords that you want to rehearse later in the reproduction site. When the playback data is over, you often remember the exact word even though you remember the word image. In this way, when searching later, it is easy to understand the timing to look for often. This can be said to be a candidate timing, so it is highly probable that at this timing there is discussion using important keywords, their synonyms, and similar nuance words. For this reason, the identification unit 362 can make the text conversion unit 361 execute the text conversion to make the voice text since the identification unit 362 is able to understand the whole discussion if the audio data here can be visualized as text preferentially. Data may be generated. In step S 511, the identifying unit 362 does not necessarily perform processing up to textification, and records the timing of intensive search, the timing of minutes and seconds from the start of recording, and the candidate timing in association with the voice data. You may just leave it alone. There is a method of recording candidate timing information as metadata at the time of creating an audio file.

  Subsequently, the document generation unit 367 adds the appearance position of the keyword identified by the identification unit 362 to the voice data and records the voice data (step S512). After step S512, the information processing device 3 returns to the main routine of FIG. 15 described above.

  In step S513, when the manual mode in which the user manually detects a specific keyword appearing in audio data is set (step S513: Yes), the audio reproduction unit 34 reproduces audio data up to a specific phrase (step S513). S514).

  Subsequently, when an instruction signal instructing repeat operation to the specific frame is input from the input unit 32 (step S515: Yes), the information processing device 3 returns to step S514 described above. On the other hand, when the instruction signal instructing the repeat operation to the specific frame is not input from the input unit 32 (step S515: No), the information processing device 3 proceeds to step S516 described later.

  In step S513, when the manual mode in which the user manually detects a specific keyword appearing in the voice data is not set (step S513: No), the information processing device 3 proceeds to step S512.

  In step S516, when there is an operation of inputting a keyword through the input unit 32 (step S516: Yes), the text conversion unit 361 transcribes the keyword according to the operation of the input unit 32 (step S517). .

  Subsequently, the document generation unit 367 adds an index to the voice data at the time when the keyword is input through the input unit 32 and records the index (step S518). After step S518, the information processing device 3 proceeds to step S512.

  In step S516, when there is no operation to input a keyword via the input unit 32 (step S516: No), the information processing device 3 proceeds to step S512.

Referring back to FIG. 15, the description of step S411 and subsequent steps will be described.
In step S411, the display control unit 366 adds an index to the appearance position where the keyword specified by the specifying unit 362 appears on the time bar displayed by the display unit 35, and causes the display unit 35 to display the index. Specifically, as shown in FIG. 18, the display control unit 366 causes the index B1 to be displayed at the appearance position where the keyword specified by the identification unit 362, for example, “confirmation” appears on the time bar T2 displayed by the display unit 35. Are displayed on the display unit 35. More specifically, the display control unit 366 sets the index B1 as the index B1 at the appearance position where the keyword specified by the specifying unit 362, for example, "confirmation" appears in the vicinity of the time bar T2 displayed by the display unit 35 (1) Are displayed on the display unit 35. Thereby, the user can intuitively grasp the appearance position of the desired keyword. The display control unit 366 causes the display unit 35 to superimpose (1) as an index B1 on the appearance position where the keyword specified by the specifying unit 362 appears on the time bar T2 displayed by the display unit 35. Alternatively, graphics and text data may be superimposed as the index B1, or the color of the time bar T2 of the appearance position may be displayed distinguishably from other areas. Of course, the display control unit 366 may cause the display unit 35 to display the time of the appearance position where the keyword specified by the specifying unit 362 appears.

  Further, as shown in FIG. 19, when the user sets three keywords, for example, when each of “confirm”, “AB company” and “delivery time” is set, the display control unit 366 causes the display unit 35 to display In the time bar T2, the display unit is added with (1), (2) and (3) as the index B1, the index B2 and the index B3 at the appearance position where each of the three keywords specified by the specifying unit 362 appears Display on 35 In this case, the display control unit 366 is an appearance position where each of the three keywords specified by the specifying unit 362 appears in a predetermined time (for example, 10 seconds or more) on the time bar T2 displayed by the display unit 35. An index may be added to the display unit 35 and displayed on the display unit 35. At this time, the display control unit 366 may add an index to the appearance position where the first keyword appears on the time bar T2 and cause the display unit 35 to display the position. As a result, it is possible to intuitively grasp the appearance position where a plurality of keywords desired by the user appear on the voice data.

  Subsequently, when either the index or each sound source (icon) on the time bar is specified via the input unit 32 (step S412: Yes), the voice control unit 365 is specified via the input unit 32. The audio data is skipped until the time corresponding to the index on the time bar or the time chart corresponding to the designated sound source, and the audio reproduction unit 34 is made to reproduce (step S413). Specifically, as shown in FIG. 20, when the user designates index (1) of arrow A via input unit 32, voice control unit 365 causes time bar T2 designated via input unit 32 to be designated. The audio data is skipped until the time corresponding to the upper index, and the audio reproduction unit 34 is made to reproduce. As a result, the user can intuitively grasp the appearance position of the desired keyword, and can write out the desired position.

  Thereafter, when an operation to end documentization is performed via the input unit 32 (step S 414: Yes), the document generation unit 367 determines the document input by the user via the input unit 32, the voice data, and the identification unit A document file in which the appearance position specified by 362 is associated is generated and recorded in the recording unit 33 (step S415). After step S415, the information processing device 3 ends the present process. On the other hand, when the operation to end the document conversion is not performed via the input unit 32 (step S414: No), the information processing device 3 returns to the above-described step S408.

  In step S412, when the index on the time bar is not designated via the input unit 32 (step S412: No), the information processing device 3 proceeds to step S414.

  In step S 416, the textification unit 361 documents the text data in accordance with the operation of the input unit 32. After step S416, the information processing device 3 proceeds to step S412.

  In step S404, when the reproduction operation for reproducing the audio data is not performed via the input unit 32 (step S404: No), the information processing device 3 ends the present process.

  In step S401, when the user does not perform the documentizing operation for making the collection while reproducing the audio data (step S401: No), the information processing device 3 performs the process according to the other mode process other than the documentizing operation. Execute (step S417). After step S417, the information processing device 3 ends the present process.

  According to the first embodiment of the present invention described above, based on the estimation result estimated by the sound source position estimation unit 295, the display control unit 303 causes the display unit 23 to provide sound source position information on each position of a plurality of sound sources. Since the display is made, the position of the speaker at the time of recording can be intuitively grasped.

  Further, according to the first embodiment of the present invention, since the display control unit 303 causes the display unit 23 to display the sound source position information based on the determination result determined by the display position determination unit 296, the shape of the display unit 23 The position of the speaker according to the user can be intuitively grasped.

  Further, according to the first embodiment of the present invention, the display position determination unit 296 determines the display position of each of the plurality of sound sources when the center of the display area of the display unit 23 is the information acquisition device 2. The position of the speaker when the acquisition device 2 is at the center can be intuitively grasped.

  Further, according to the first embodiment of the present invention, the display control unit 303 causes the display unit 23 to display a plurality of pieces of sound source information generated by the sound source information generation unit 298 as sound source position information. Intuitively grasp the gender and number of persons.

  Further, according to the first embodiment of the present invention, the sound file generation unit 302 generates sound data, sound source position information estimated by the sound source position estimation unit 295, and a plurality of sound source information generated by the sound source information generation unit 298. , Position information on the position of appearance specified by the sound specifying unit 299, position of the index added by the index adding unit 301, or time information on time on voice data added by the index; Since the voice file in which the voice text data is associated is generated and recorded in the voice file recording unit 262, it is possible to grasp the position desired by the creator when the information processing device 3 creates a collection.

  Further, according to the first embodiment of the present invention, the sound source information generation unit 298 adds information indicating that the sound source has moved to the sound source information of the sound source determined to be moving by the movement determination unit 300. It is possible to intuitively grasp the speaker who moved from time to time.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the first embodiment described above, the positions of a plurality of sound sources are estimated and displayed on the display unit 23 based on the two audio data generated by each of the first sound collecting unit 20 and the second sound collecting unit 21. However, in the second embodiment, the positions of a plurality of sound sources are estimated and displayed by further using voice data generated by the external microphone inserted into the external input detection unit 22. Hereinafter, the configuration of the information acquisition device according to the second embodiment will be described. In addition, the same code | symbol is attached | subjected to the structure same as the information acquisition apparatus 2 which concerns on Embodiment 1 mentioned above, and description is abbreviate | omitted.

  FIG. 21 is a perspective view showing a schematic configuration before an external microphone is attached to the information acquisition apparatus according to the second embodiment of the present invention. FIG. 22 is a perspective view showing a schematic configuration after an external microphone is attached to the information acquisition apparatus according to the second embodiment of the present invention.

  As shown in FIGS. 21 and 22, in the information acquisition device 2, the plug 101 of the external microphone 100 is inserted into the external input detection unit 22. The external microphone 100 has a rectangular shape, and the third sound collecting unit 102 and the fourth sound collecting unit 103 are provided on the side surface. The external microphone 100 is inserted in a direction perpendicular to the longitudinal direction of the upper surface of each of the first sound collecting unit 20 and the second sound collecting unit 21 provided in the information acquisition device 2 and is detachable from the information acquisition device 2 Attached to The external microphone 100 is configured using any one of microphones such as a unidirectional microphone, a nondirectional microphone and a bidirectional microphone, and a stereo microphone capable of collecting left and right sounds. In the second embodiment, a case will be described in which a stereo microphone configured of the third sound collecting unit 102 and the fourth sound collecting unit 103 as the external microphone 100 is used. In addition, external microphones 100 have different frequency characteristics from the built-in microphones (the first sound collecting unit 20 and the second sound collecting unit 21), or those with good performance, or are separated from the information acquisition device 2 by an extension cable. It may be used in a position, attached to the collar of the user, or the like.

  The external input detection unit 22 generates two audio data (third audio data and fourth audio data) collected by the left and right microphones, respectively, and outputs the audio data to the device control unit 29.

  Thus, in the information acquisition device 2 into which the external microphone 100 is inserted, the sound source position estimation unit 295 generates a plurality of audio data generated by each of the first sound collection unit 20, the second sound collection unit 21 and the external microphone 100. Use to estimate the position of multiple sound sources. Specifically, as shown in FIG. 23, when the sound sources A1 and A2 are located at the same distance D1, the sound source position estimating unit 295 is used for each of the first sound collecting unit 20 and the second sound collecting unit 21. The arrival time differences reached by the audio signals become equal, and it becomes difficult to estimate the positions of the sound sources A1 and A2. Therefore, the sound source position estimating unit 295 inserts the external microphone 100 in the direction orthogonal to the longitudinal direction of the upper surface of each of the first sound collecting unit 20 and the second sound collecting unit 21 provided in the information acquisition device 2. As shown in FIG. 24, the difference between the arrival time for the audio signal to reach each of the first sound collecting unit 20 and the second sound collecting unit 21 and the arrival time for the audio signal to reach the external microphone 100 as shown in FIG. As a result, the position of each sound source in the depth direction can be estimated using a plurality of sound data generated by each of the first sound collection unit 20, the second sound collection unit 21, and the external microphone 100.

  According to the second embodiment of the present invention described above, the position of the speaker at the time of recording can be intuitively grasped.

  Further, according to the second embodiment of the present invention, the external microphone 100 is orthogonal to the longitudinal direction of the upper surface of each of the first sound collecting unit 20 and the second sound collecting unit 21 provided in the information acquisition device 2. Since the insertion time in the direction causes a difference between the arrival time for the audio signal to reach each of the first sound collecting unit 20 and the second sound collecting unit 21 and the arrival time for the audio signal to reach the external microphone 100, The sound source position estimation unit 295 can estimate the position of each sound source in the depth direction using a plurality of audio data generated by each of the first sound collection unit 20, the second sound collection unit 21, and the external microphone 100.

  Further, although the information acquisition device and the information processing apparatus according to the present invention transmit and receive data bidirectionally via the communication cable, without being limited to this, the information processing apparatus transmits information via a server or the like. The processing device may acquire an audio file storing audio data generated by the information acquisition device, or the information acquisition device may transmit the audio file storing audio data to a server on the network.

  In addition, although the information processing apparatus according to the present invention is obtained by receiving an audio file storing audio data from an information acquisition device, the information processing apparatus is not limited to this, and is not limited to this. May be acquired.

  In the description of the flowchart in the present specification, the context of processing between steps is clearly indicated using expressions such as "first", "after", "following", etc., in order to implement the present invention. The order of processing required is not uniquely determined by their representation. That is, the order of processing in the flowcharts described herein can be changed without contradiction. Further, not limited to such a program comprising simple branch processing, more judgment items may be comprehensively judged and branched. In that case, artificial intelligence techniques may be used in combination, such as machine learning as the user is prompted to perform manual operations and learning is repeated. Further, it is also possible to learn operation patterns performed by many experts and perform deep learning by inserting more complicated conditions.

  Thus, the present invention can include various embodiments not described herein, and various design changes can be made within the scope of the technical idea specified by the claims. It is.

1: Transcriber system 2: Information acquisition device 3: Information processing device 4: Communication cable 20: First sound collection unit 21: Second sound collection unit 22: external input detection unit; 23: display unit; 24: clock; 25: input unit; 26: recording unit; 27: communication unit; 28: output unit 29: device control unit 31: communication unit 32: input unit 33: recording unit 34: audio reproduction unit 35: display unit 36: information Processing control unit; 100: external microphone; 101: plug; 102: third sound collecting unit; 103: fourth sound collecting unit: 251: touch panel; 261: program recording unit 262: voice file recording unit; 291: signal processing unit; 292: text conversion unit; 293: text Sound control unit; 294: sound source position estimation unit; 296 ... display position determination unit; 297 ... voice print judgment unit; 298 ... sound source information generation unit; · · · · · · Voice identification unit; 300 · · · movement determination unit 301 · · · · · · · · · · · · · · · · · · · · · · · · · · · voice identification unit; · Speech-to-text dictionary data storage unit 361 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · keyword determination unit; · · Display control unit; 367 · · · Document generation unit

Claims (12)

A display unit capable of displaying an image;
A plurality of sound collection units provided at mutually different positions and collecting sound emitted from each of the plurality of sound sources to generate sound data;
A sound source position estimation unit configured to estimate the position of each of the plurality of sound sources based on the voice data generated by each of the plurality of sound collection units;
A display control unit that causes the display unit to display sound source position information on each position of the plurality of sound sources based on the estimation result estimated by the sound source position estimating unit;
An information acquisition apparatus comprising: A display position determination unit is further provided that determines the display position of each of the plurality of sound sources in the display area of the display unit based on the shape of the display area of the display unit and the estimation result estimated by the sound source position estimation unit. ,
The information acquisition apparatus according to claim 1, wherein the display control unit causes the display unit to display the sound source position information based on the determination result determined by the display position determination unit. A voiceprint determination unit that determines the volume of each speaker emitted by each of a plurality of speakers based on the voice data;
A sound source information generation unit that generates, as sound source information, an icon schematically indicating each of the plurality of speakers based on comparison of the volume of each of the plurality of speakers determined by the voiceprint determination unit;
The information acquisition apparatus according to claim 2, further comprising:   Sound source information for generating sound source information in which icons schematically showing speakers are made different from each other based on the voice data and the length and volume of voice of each speaker uttered by each of a plurality of speakers The information acquisition apparatus according to claim 2, further comprising a generation unit.   The information acquisition device according to claim 2, wherein the display position determination unit determines the display position when the center of the display area of the display unit is the information acquisition device. A voiceprint determination unit that determines a voiceprint of each of the plurality of sound sources based on the voice data;
A sound source information generation unit configured to generate a plurality of pieces of sound source information related to each of the plurality of sound sources based on the determination result determined by the voiceprint determination unit;
Equipped with
6. The information acquisition apparatus according to claim 5, wherein the display control unit causes the display unit to display the plurality of sound source information as the sound source position information. A voice identification unit that identifies an appearance position where each voiceprint determined by the voiceprint determination unit appears on the voice data;
An audio file generation unit that generates an audio file in which the audio data, the sound source position information, the plurality of sound source information, and the appearance position are associated with one another and is recorded in a recording medium.
The information acquisition apparatus according to claim 6, comprising: The apparatus further comprises a movement determination unit that determines whether each of the plurality of sound sources is moving based on the estimation result estimated by the sound source position estimation unit and the determination result determined by the voice print determination unit.
8. The information acquisition according to claim 7, wherein the sound source information generation unit adds information indicating that the sound source has moved to the sound source information of the sound source determined to be moving by the movement determination unit. machine.   The information acquisition device according to any one of claims 1 to 8, wherein the plurality of sound collection units are detachable with respect to the information acquisition device.   The information acquisition device according to any one of claims 1 to 9, further comprising an external microphone that is detachable to the information acquisition device. Information acquisition comprising: a display unit capable of displaying an image; and a plurality of sound collection units provided at mutually different positions and collecting sound emitted from each of a plurality of sound sources to generate sound data The display method executed by the device, and
A sound source position estimation step of estimating the positions of the plurality of sound sources based on the voice data generated by each of the plurality of sound collection units;
A display control step of causing the display unit to display sound source position information on each position of the plurality of sound sources based on the estimation result estimated in the sound source position estimation step;
A display method characterized in that Information acquisition comprising: a display unit capable of displaying an image; and a plurality of sound collection units provided at mutually different positions and collecting sound emitted from each of a plurality of sound sources to generate sound data To the equipment
A sound source position estimation step of estimating the positions of the plurality of sound sources based on the voice data generated by each of the plurality of sound collection units;
A display control step of causing the display unit to display sound source position information on each position of the plurality of sound sources based on the estimation result estimated in the sound source position estimation step;
A program characterized by causing

Images