JP6392578B2 - Audio processing apparatus, audio processing method, and audio processing program - Google Patents

Description

The present invention is the invention, the audio processing device, voice processing method, and a voice processing program.

In recent years, IC recorders capable of recording (recording) and reproducing voices at the time of a meeting have been put into practical use (for example, see Patent Document 1).
Specifically, the IC recorder described in Patent Document 1 converts voice input via a microphone into voice data (digital data), and then records the voice data in a memory. In addition, the IC recorder converts audio data recorded in the memory into an audio signal (analog signal), and then outputs (reproduces) audio based on the audio signal via a speaker.

Such an IC recorder generally displays a reproduction screen as shown below when reproducing audio.
Specifically, the playback screen is provided with a time bar corresponding to the time from the start to the end of voice recording and a slider arranged on the time scale and a slider indicating the playback position. It is a screen.
That is, the user of the IC recorder can grasp the reproduction position of the already recorded audio data by confirming the reproduction screen (time bar) when reproducing the audio.

JP 2012-205086 A

However, the conventional playback screen is only provided with a time bar. For this reason, if the user does not check the actually reproduced voice, for example, the user can grasp the situation at the time of recording such as who the speaker is and the state of the speaker's tension. There is a problem that it is not possible.
Therefore, there is a demand for a technique that can grasp the situation at the time of recording from the playback screen and can improve convenience.

The present invention was made in view of the above, an object of speech processing apparatus can be improved convenience, voice processing method, and to provide a voice processing program.

In order to solve the above-described problems and achieve the object, an audio processing apparatus according to the present invention analyzes an audio data acquisition unit that acquires audio data, and analyzes the audio data, and analyzes the audio included in the audio data. Among them, an audio data analysis unit for discriminating a component with high tension is used to generate a reproduction screen of the audio data by associating the high tension component with the time during which the high tension component is included in the audio data. A reference information generation unit that generates reference information to be generated , and the sound data analysis unit analyzes the sound data and generates sound included in the sound data for each predetermined time range in the sound data. When the speaker cannot be specified when the speaker is identified and the high tension component is identified and the speaker cannot be identified Range of sound, characterized that you assumed the speaker identified in the immediately preceding or time range immediately after emitted for that time range.

Further, the audio processing method according to the present invention is an audio processing method performed by an audio processing apparatus, wherein an audio data acquisition step for acquiring audio data, and the audio data are analyzed and included in the audio included in the audio data, A reference used when generating a reproduction screen of the audio data by associating the audio data analysis step for discriminating the high tension component with the time during which the high tension component and the high tension component of the audio data are included look including a reference information generating step of generating information, in the audio data analyzing step analyzes the voice data for each predetermined time range in the audio data, it issues a voice included in the voice data If the speaker is identified and the high tension component is determined and the speaker cannot be identified, Speech time range that can not be identified, and estimates as the speaker identified in the immediately preceding or time range immediately after emitted for that time range.

  A speech processing program according to the present invention causes a speech processing apparatus to execute the speech processing method described above.

According to the audio processing device of the present invention, an image based on the height of the speaker's tension is displayed on the reproduction screen without confirming the actually reproduced audio, so that the speaker can be reproduced from the reproduction screen. You can grasp the situation when recording.

FIG. 1 is a block diagram showing a configuration of an electronic apparatus according to Embodiment 1 of the present invention. FIG. 2 is a flowchart showing the operation of the electronic device shown in FIG. FIG. 3 is a flowchart showing the speaker display process (step S111) shown in FIG. FIG. 4 is a diagram illustrating an example of a situation where the first and second audio data to be subjected to the speaker display process (step S111) illustrated in FIGS. 2 and 3 are generated (recorded). FIG. 5 is a diagram illustrating an example of reference information generated when the speaker display process (step S111) is executed on the first and second audio data generated in the situation of FIG. 6 is a diagram showing an example of a speaker display reproduction screen generated based on the reference information shown in FIG. FIG. 7 is a diagram showing a modification of the first embodiment of the present invention. FIG. 8 is a block diagram showing a configuration of a speech processing system according to Embodiment 2 of the present invention. FIG. 9 is a flowchart showing speaker display processing (step S111) according to Embodiment 2 of the present invention. FIG. 10 is a flowchart showing the operation of the server shown in FIG. FIG. 11A is a diagram showing a modification of the speaker display reproduction screen described in the first and second embodiments of the present invention. FIG. 11B is a diagram showing a modification of the speaker display reproduction screen described in the first and second embodiments of the present invention. FIG. 11C is a diagram showing a modification of the speaker display reproduction screen described in the first and second embodiments of the present invention. FIG. 12 is a diagram showing a modification of the reference information described in the first and second embodiments of the present invention. FIG. 13 is a diagram showing an example of a speaker display reproduction screen generated based on the reference information shown in FIG.

  DESCRIPTION OF EMBODIMENTS Hereinafter, modes for carrying out the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings. The present invention is not limited to the embodiments described below. Furthermore, the same code | symbol is attached | subjected to the same part in description of drawing.

(Embodiment 1)
[Configuration of electronic equipment]
FIG. 1 is a block diagram showing a configuration of electronic apparatus 1 according to Embodiment 1 of the present invention.
The electronic device 1 is configured as an IC recorder, a digital camera, a digital video camera, a mobile phone, a tablet mobile device, or the like. And the electronic device 1 discriminate | determines the characteristic component (speaker's tension) of the said voice by analyzing the audio | voice data containing the audio | voice which the speaker uttered, and clarified the time when the said characteristic component produced with the time bar. Display the playback screen.
Hereinafter, the main part of the present invention will be mainly described as the configuration of the electronic apparatus 1. As shown in FIG. 1, the electronic device 1 includes a first audio data generation unit 11, a second audio data generation unit 12, an operation unit 13, a display unit 14, a clock unit 15, and a memory unit 16. A recording unit 17, an audio output unit 18, and a device-side control unit 19.

The first sound data generation unit 11 generates first sound data based on the input sound under the control of the device-side control unit 19. As shown in FIG. 1, the first audio data generation unit 11 includes a first microphone 111, a first amplifier 112, and a first A / D conversion unit 113.
The first microphone 111 inputs sound and converts it into an electrical signal. Here, the first microphone 111 is disposed on the upper left side when the electronic device 1 is viewed from the front (see FIG. 4).
The first amplifier 112 receives an electrical signal from the first microphone 111, and performs predetermined analog processing (noise reduction processing for reducing noise components, increasing gain to maintain a constant output level for the electrical signal. Gain processing, etc.) and output to the first A / D converter 113.
The first A / D conversion unit 113 receives the electrical signal from the first amplifier 112 and performs A / D conversion on the electrical signal to convert the electrical signal into a digital signal (first audio data). To the side controller 19.

Similar to the first sound data generation unit 11, the second sound data generation unit 12 generates second sound data based on the input sound under the control of the device-side control unit 19. As shown in FIG. 1, the second audio data generation unit 12 is similar to the first microphone 111, the first amplifier 112, and the first A / D conversion unit 113 that constitute the first audio data generation unit 11, respectively. 2 microphones 121, a second amplifier 121, and a second A / D converter 123.
Here, the second microphone 121 is arranged on the upper right side (side facing the first microphone 111) when the electronic device 1 is viewed from the front (see FIG. 4).

The operation unit 13 is configured by using a button, a switch, a touch panel, or the like that receives a user operation, and outputs an instruction signal corresponding to the user operation to the device side control unit 19.
And the operation part 13 has a function as an operation reception part which concerns on this invention.
The display unit 14 is configured using a display panel made of liquid crystal, organic EL (Electro Luminescence), or the like. The display unit 14 displays an image such as a speaker display reproduction screen under the control of the device-side control unit 19.
The clock unit 15 has a function of generating date and time information (hereinafter referred to as a time stamp) related to the date and time when the audio data is generated by the first and second audio data generation units 11 and 12 in addition to the timekeeping function. The time stamp generated by the clock unit 15 is output to the device-side control unit 19.

The memory unit 16 temporarily stores the first and second audio data generated by the first and second audio data generation units 11 and 12 and information being processed by the device-side control unit 19, respectively.
The recording unit 17 stores various programs (including audio processing programs) executed by the device-side control unit 19 and the first and second audio data generated by the first and second audio data generation units 11 and 12, respectively. Record. The recording unit 17 records the reference information generated by the device-side control unit 19 in association with the corresponding first and second audio data under the control of the device-side control unit 19.

The audio output unit 18 outputs audio based on the first and second audio data recorded in the recording unit 17 under the control of the device-side control unit 19. As shown in FIG. 1, the audio output unit 18 includes a D / A conversion unit 181, an amplifier 182, and a speaker 183.
The D / A conversion unit 181 performs D / A conversion on the first and second audio data recorded in the recording unit 17 to convert each of the first and second audio data into an analog signal, and the sum signal of each analog signal. Is output to the amplifier 182.
The amplifier 182 receives the audio signal (sum signal) from the D / A converter 181, performs predetermined analog processing on the audio signal, amplifies the audio signal, and outputs the audio signal to the speaker 183.
The speaker 183 receives the audio signal from the amplifier 182 and outputs audio based on the audio signal.

  The device-side control unit 19 is configured using a CPU (Central Process Unit) or the like, and performs instructions and data transfer corresponding to each unit constituting the electronic device 1 in response to an instruction signal from the operation unit 13 or the like. The operation of the electronic device 1 is comprehensively controlled. As shown in FIG. 1, the device-side control unit 19 includes an audio data acquisition unit 191, an audio data analysis unit 192, a reference information generation unit 193, a playback screen generation unit 194, a display control unit 195, an audio And a control unit 196.

The audio data acquisition unit 191 performs the following functions when the electronic device 1 is set to the recording mode in response to a user operation (such as operation of a mode switch) on the operation unit 13 by the user.
The voice data acquisition unit 191 generates first and second voice data in the first and second voice data generation units 11 and 12 in response to a recording start operation (such as pressing a recording switch) to the operation unit 13 by the user. The first and second audio data are acquired. Then, the audio data acquisition unit 191 associates the time stamp generated by the clock unit 15 (date and time information regarding the generation date and time of the first and second audio data) with the first and second audio data and stores them in the memory unit 16. Store sequentially. Also, the audio data acquisition unit 191 sends the first and second audio data to the first and second audio data generation units 11 and 12 in response to a recording end operation (such as pressing a stop switch) to the operation unit 13 by the user. And the first and second audio data (including the time stamp) stored in the memory unit 16 are recorded in the recording unit 17.

The audio data analysis unit 192 is set so that the electronic device 1 is set to a reproduction mode (a mode for reproducing the first and second audio data) in response to a user operation (such as an operation of a mode switch) on the operation unit 13 by the user. The first and second audio data recorded in the recording unit 17 are analyzed. As shown in FIG. 1, the audio data analysis unit 192 includes an object specifying unit 1921 and a feature component determination unit 1922.
The object specifying unit 1921 analyzes the first and second sound data to specify the speaker who has emitted the sound included in the first and second sound data.
The feature component determination unit 1922 analyzes the first and second sound data to determine the sound feature component (speaker's tension) included in the first and second sound data.

The reference information generation unit 193 performs the following functions when the electronic device 1 is set to the playback mode.
The reference information generation unit 193 includes a speaker identified by the object identification unit 1921, a speaker tension determined by the feature component determination unit 1922, and a timestamp indicating the date and time when the voice of the speaker is included. Reference information used when generating the reproduction screens of the first and second audio data is generated. Here, the speaker's tension (speech feature) is assumed to be emotionally high, but the concentration of the story (for example, one speaker explains and others quietly listen to it) Etc.) may be reflected. In this case, the speaker's tension is detected by detecting the relative loudness of the voices of multiple people detected, the degree of the pace of the words (speaking to speak) and the speed (speaking). May be determined. In other words, the tension of a speaker can detect relative changes over time of one speaker's voice, can be judged by absolute numerical data, It is determined by detection.
And the audio | voice data acquisition part 191, the audio | voice data analysis part 192, and the reference information production | generation part 193 mentioned above have a function as an audio | voice processing apparatus which concerns on this invention.

The reproduction screen generation unit 194 generates a speaker display reproduction screen when the display flag for speaker display is turned on in response to a user operation on the operation unit 13 by the user. Note that the display flag of the speaker display is stored in the memory unit 16.
Specifically, the playback screen generation unit 194 arranges a time bar indicating the playback position, and at each time corresponding to the time bar based on the reference information generated by the reference information generation unit 193, the time ( A speaker display reproduction screen in which an identification image for identifying the speaker associated with the (time stamp) and the tension of the speaker is generated. Here, the speaker's tension (speech feature) is assumed to be emotionally high, but the concentration of the story (for example, one speaker explains and others quietly listen to it) Etc.) may be reflected. In this case, the speaker's tension is detected by detecting the relative loudness of the voices of multiple people detected, the degree of the pace of the words (speaking to speak) and the speed (speaking). May be determined. In other words, the tension of a speaker can detect relative changes over time of one speaker's voice, can be judged by absolute numerical data, It is determined by detection.
In addition, the playback screen generation unit 194 arranges only the time bar indicating the playback position when the display flag of the speaker display is turned off in response to the user operation on the operation unit 13 by the user (described above). A normal playback screen (without an identification image) is generated.

The display control unit 195 displays a selection screen for allowing the user to select the first and second audio data to be reproduced, a speaker display reproduction screen generated by the reproduction screen generation unit 194, a normal reproduction screen, and the like on the display unit 14. Display.
The audio control unit 196 performs the following functions when the electronic device 1 is set to the playback mode.
The audio control unit 196 controls the operation of the audio output unit 18 in response to a user's reproduction start operation (such as pressing a reproduction switch) on the operation unit 13, and the first and second audio recorded in the recording unit 17. Start outputting audio based on the data. In addition, the voice control unit 196 causes the voice output unit 18 to end the voice output in response to a reproduction end operation (such as pressing a stop switch) to the operation unit 13 by the user.

[Operation of electronic equipment]
Next, the operation of the electronic device 1 described above will be described.
FIG. 2 is a flowchart showing the operation of the electronic device 1.
When the electronic device 1 is turned on by the user operating the operation unit 13 (step S101: Yes), the device-side control unit 19 determines whether or not the electronic device 1 is set to the recording mode ( Step S102).

When it is determined that the recording mode is not set (step S102: No), the electronic apparatus 1 proceeds to step S107.
On the other hand, when it is determined that the recording mode is set (step S102: Yes), the device-side control unit 19 determines whether or not the user has performed a recording start operation on the operation unit 13 (step S103). .

When it is determined that there is no recording start operation (step S103: No), the electronic device 1 returns to step S101.
On the other hand, if it is determined that there has been a recording start operation (step S103: Yes), the audio data acquisition unit 191 generates the first and second audio data in the first and second audio data generation units 11 and 12. Start (Recording). In addition, the clock unit 15 starts generating (clocking) a time stamp. Then, the audio data acquisition unit 191 sequentially stores the time stamp in the memory unit 16 in association with the first and second audio data (step S104).

Subsequently, the device-side control unit 19 determines whether or not the user has performed a recording end operation on the operation unit 13 (step S105).
When it is determined that there is no recording end operation (step S105: No), the electronic device 1 continues recording and timing.
On the other hand, when it is determined that the recording end operation has been performed (step S105: Yes), the audio data acquisition unit 191 generates the first and second audio data in the first and second audio data generation units 11 and 12. End. In addition, the clock unit 15 ends the time stamp generation. Then, the audio data acquisition unit 191 records the first and second audio data (including the time stamp) stored in the memory unit 16 in the recording unit 17 (step S106). Thereafter, the electronic device 1 returns to step S101.
Steps S103 to S106 described above correspond to the audio data acquisition step according to the present invention.

If it is determined in step S102 that the recording mode is not set (step S102: No), the device-side control unit 19 determines whether or not the electronic device 1 is set to the playback mode (step S107). .
If it is determined that the playback mode is not set (step S107: No), the electronic apparatus 1 proceeds to step S118.
On the other hand, when it is determined that the playback mode is set (step S107: Yes), the display control unit 195 displays a selection screen on the display unit 14 (step S108).
Here, the selection screen is a screen that allows the user to select a plurality of first and second audio data recorded in the recording unit 17. For example, each selection screen is associated with each of the plurality of first and second audio data. It is a screen in which each date based on the time stamp is displayed in a list.

Subsequently, the device-side control unit 19 determines whether or not the user has performed a selection operation (an operation for selecting any of the first and second audio data in the selection screen) on the operation unit 13 (step S109). .
When it is determined that there is no selection operation (step S109: No), the electronic device 1 continues to display the selection screen.
On the other hand, when it is determined that the selection operation has been performed (step S109: Yes), the device-side control unit 19 determines whether or not the display flag of the speaker display is on (step S110).

If it is determined that the display flag for speaker display is on (step S110: Yes), the electronic device 1 executes speaker display processing for generating and displaying a speaker display reproduction screen (step S111). ).
The details of the speaker display process will be described later.
On the other hand, when it is determined that the display flag of the speaker display is in the off state (step S110: No), the playback screen generation unit 194 generates a normal playback screen. Then, the display control unit 195 displays the normal playback screen on the display unit 14 (step S112).

After step S111 or step S112, the device-side control unit 19 determines whether or not the user has performed a reproduction start operation on the operation unit 13 (step S113).
If it is determined that there is no reproduction start operation (step S113: No), the electronic device 1 proceeds to step S117.
On the other hand, when it is determined that there has been a reproduction start operation (step S113: Yes), the audio control unit 196 records the first and second audio data selected by the selection operation by the user (step S109). Read from. Then, the sound control unit 196 causes the sound output unit 18 to start outputting (reproducing) sound based on the first and second sound data (step S114).

Subsequently, the device-side control unit 19 determines whether or not the user has performed a reproduction end operation on the operation unit 13 (step S115).
When it is determined that there is no reproduction end operation (step S115: No), the electronic device 1 continues the reproduction.
On the other hand, when it is determined that the reproduction end operation has been performed (step S115: Yes), the audio control unit 196 causes the audio output unit 18 to end the audio output (reproduction) (step S116). Note that, as a result of continuing the reproduction in step S115, when all the first and second audio data have been reproduced, the process proceeds to step S116.

After step S116 or when it is determined in step S113 that there is no reproduction start operation (step S113: No), the device-side control unit 19 performs the reproduction target (first and first) to the operation unit 13 by the user. It is determined whether or not there has been a change operation of (2 audio data) (step S117).
When it is determined that there is no reproduction target change operation (step S117: No), the electronic device 1 returns to step S113.
On the other hand, if it is determined that there has been an operation to change the playback target (step S117: Yes), the electronic device 1 returns to step S101, goes through steps S101, S102, and S107, and then selects again in step S108. Display the screen.

  If it is determined in step S107 that the playback mode is not set (step S107: No), the electronic device 1 executes another process different from the process described above (step S118). Thereafter, the electronic device 1 returns to step S101.

[Speaker display processing]
Next, the speaker display process (step S111) described above will be described.
FIG. 3 is a flowchart showing the speaker display process (step S111).
The device-side control unit 19 determines whether or not reference information of the first and second voice data (first and second voice data selected in step S109) to be subjected to speaker display processing has already been generated. (Step S111A). That is, in step S111A, the device-side control unit 19 determines whether or not reference information is associated with the first and second audio data recorded in the recording unit 17.
When it is determined that the reference information has been generated (step S111A: Yes), the electronic device 1 proceeds to step S111P.
On the other hand, when it is determined that the reference information has not yet been generated (step S111A: No), the target object specifying unit 1921 selects one period (first period) in the first and second audio data to be subjected to the speaker display process ( For example, data corresponding to 5 seconds is read (step S111B).
Hereinafter, data corresponding to one period in the first audio data is referred to as a first data element, and data corresponding to one period in the second audio data is referred to as a second data element.

Subsequently, the object specifying unit 1921 analyzes the first and second data elements corresponding to the one period (hereinafter referred to as the corresponding period) read out in step S111B, thereby including the first and second data elements. The speaker who has spoken is specified (step S111C).
Specifically, the object specifying unit 1921 specifies the direction of the speaker with respect to the electronic device 1 by comparing the volume of each voice included in the first and second data elements corresponding to the corresponding period. In addition, the object specifying unit 1921 specifies the gender of the speaker based on the audio frequency included in the first and second data elements. Since the frequency of pronunciation such as vowels is higher than that of men, it can be used for sex determination. In addition, gender can be determined by words, contents, intonation used, and the like. When there are male and female speakers, the voices may be compared to determine the sex, or the sex may be determined based on whether the frequency is higher or lower than the specific frequency. Furthermore, gender can be determined based on the characteristics of words used, sentences, and endings. In addition, gender may be determined based on the similarity between the model voices of male and female. Needless to say, it is possible to determine whether the age is high or low based on the same concept. For speakers who frequently appear, there is a method of specifying by voice collation with a database registered in advance.

In step S111C, the speaker identification is not limited to the method for identifying the speaker direction and gender as described above, and the speaker may be identified as follows.
Identification data (such as user names) for identifying a plurality of users is associated with voice print data relating to the voice print of the user, and the related information is recorded in the recording unit 17 in advance. Then, the object specifying unit 1921 refers to the information recorded in the recording unit 17 and specifies voiceprint data that matches the voiceprint of the voice included in the first and second data elements. Identify identification data (user name, etc.) associated with the data.

After step S111C, the object specifying unit 1921 determines whether or not the speaker has been specified in step S111C (both the direction and gender of the speaker have been specified) (step S111D). ).
If it is determined that the speaker could not be specified (at least one of the speaker direction and gender could not be specified) (step S111D: No), the electronic device 1 The process proceeds to step S111G.

On the other hand, when it is determined that the speaker can be specified (both the direction and gender of the speaker can be specified) (step S111D: Yes), the feature component determination unit 1922 By analyzing the first and second data elements corresponding to the period, the tension of the speaker who has produced the speech included in the first and second data elements is determined for each speaker (step S111E: voice data analysis). Step).
Specifically, the feature component determination unit 1922 determines the speaker's tension based on the volume of each voice included in the first and second data elements corresponding to the corresponding period. That is, the feature component determination unit 1922 determines the speaker's tension as “high tension” when the volume of the speaker's voice is greater than or equal to a predetermined first threshold value compared to the volume of the previous period. In other cases, the speaker's tension is determined as “normal”. The prosodic features of speech change depending on emotions, but this can be analyzed by high / low, strong / weak voice, rhythm / tempo, fundamental frequency, power, duration, etc. It is said that there is a relationship between the degree of emotion contained in the voice representing emotion and the fundamental frequency pattern, and it is possible to analyze even the pitch frequency / amplitude change pattern. In addition, accents, included words, exclamations, and the like may be detected, and “high tension” can be determined by combining these detection results or using one of them. As will be described later, a laughing voice or a roaring voice may be analyzed. This can also be determined by pattern matching of voice (voice print data). The speaker's tension (speech feature) need not be limited to intense emotional highs such as emotions, but the concentration of the story (for example, one speaker explains it to other people Listening quietly, etc.) may be reflected. In this case, the speaker's tension is detected by detecting the relative loudness of the voices of multiple people detected, the degree of the pace of the words (speaking to speak) and the speed (speaking). May be determined. In other words, the tension of a speaker can detect relative changes over time of one speaker's voice, can be judged by absolute numerical data, It is determined by detection.

Note that the determination of the speaker's tension in step S111E is not limited to the method of determining based on the volume described above, and the tension may be determined as follows.
For example, the feature component determination unit 1922 determines the speaker's tension based on the frequency of speech included in the first and second data elements. Specifically, the feature component determination unit 1922 sets the speaker's tension to “high tension” when the frequency of the speaker's voice is higher than a predetermined second threshold value compared with the frequency of the voice in the immediately preceding period. In other cases, the speaker's tension is determined as “normal”. “High tension” may reflect not only high emotion of the speaker such as emotions but also concentration of the talk. In this case, the tension is increased by detecting the relative loudness of the detected voices of multiple people, the degree of the pace of the words (speaking to speak), the speed (speaking), etc. It may be determined. In other words, it is possible to detect relative changes over time of a single speaker's voice, to make judgments based on absolute numerical data, or to detect relative differences among multiple people's voices to obtain predetermined characteristics. When a typical result is obtained, it may be determined as “high tension”.
Further, for example, the feature component determination unit 1922 determines the speaker's tension based on the time density of the phoneme component of the speech included in the first and second data elements. Specifically, the feature component discriminating unit 1922 speaks when the time density of the phoneme component of the speaker's voice is greater than or equal to a predetermined third threshold value compared to the time density of the phoneme component of the voice in the immediately preceding period. The speaker's tension is determined as “high tension”, and in other cases, the speaker's tension is determined as “normal”.
Furthermore, for example, voice print data relating to a voice print such as a laughing voice or an angry voice is recorded in the recording unit 17 in advance. Then, the feature component determination unit 1922 refers to the voiceprint data recorded in the recording unit 17, and the voice included in the first and second data elements matches the voiceprint such as a laughing voice or an angry voice based on the voiceprint data. When there is a voiceprint to be played, the speaker's tension is determined as “high tension”, and in other cases, the speaker's tension is determined as “normal”.

  After step S111E, the reference information generation unit 193 corresponds to the speaker (direction and gender) specified in step S111C, the speaker tension determined in step S111E, and the corresponding period as reference information for the corresponding period. Reference information (a “multi-speaker period” flag and a “noisy period” flag to be described later are in an off state) associated with a time stamp (generated by the clock unit 15) to be generated is generated (step S111F: reference information generation step). Then, the reference information generation unit 193 stores the generated reference information in the memory unit 16. Thereafter, the electronic device 1 proceeds to step S111J.

If it is determined in step S111D that the speaker could not be specified (at least one of the speaker direction and gender could not be specified) (step S111D: No), the target object is specified. The unit 1921 determines whether or not there are a plurality of speakers based on the direction or gender of the speakers that can be identified (step S111G).
When it is determined that there are a plurality of speakers (step S111G: Yes), the reference information generation unit 193 uses the direction or gender of the speakers that can be specified in step S111C as the reference information for the corresponding period. In addition to associating a time stamp corresponding to the corresponding period, reference information with the “multiple speaker period” flag turned on is generated (step S111H). Then, the reference information generation unit 193 stores the generated reference information in the memory unit 16. Thereafter, the electronic device 1 proceeds to step S111J.
Here, the “multi-speaker period” flag (ON state) is a flag indicating that a speaker in the corresponding period cannot be specified and that there are a plurality of speakers in the corresponding period.

On the other hand, when it is determined that there are not a plurality of speakers (step S111G: No), the reference information generation unit 193 uses the direction of the speaker that can be identified in step S111C as the reference information of the corresponding period or In addition to associating the gender with the time stamp corresponding to the relevant period, reference information with the “noisy period” flag turned on is generated (step S111I). Then, the reference information generation unit 193 stores the generated reference information in the memory unit 16. Thereafter, the electronic device 1 proceeds to step S111J.
Here, the “noisy period” flag is a flag indicating that a speaker in the corresponding period cannot be specified and that there are not a plurality of speakers in the corresponding period.
When it is determined in step S111G that there are not a plurality of speakers (step S111G: No), in addition to the case where it is determined that there is only one speaker, both the direction and gender of the speaker are determined in step S111C. It is also possible to determine whether there is a plurality of speakers or a single speaker.

After step S111F, step S111H, or step S111I, the device-side control unit 19 determines whether or not reference information has been generated in all periods in the first and second audio data to be subjected to speaker display processing. (Step S111J).
If it is determined that the reference information has not been generated in all periods (step S111J: No), the electronic device 1 returns to step S111B, and the first corresponding to the other period in the first and second audio data. First, the second data element is read, and reference information for the other period is generated.

On the other hand, when it is determined that the reference information has been generated in all periods (step S111J: Yes), the object specifying unit 1921 executes the following process (step S111K).
In step S111K, the object specifying unit 1921 includes reference information in which the “multi-speaker period” flag or the “noisy period” flag is on in the reference information of each period stored in the memory unit 16 (step S111K). It is determined whether there is reference information generated in S111H or step S111I.
When it is determined that there is no reference information in which the “multiple speaker period” flag or the “noisy period” flag is on (step S111K: No), the electronic device 1 proceeds to step S111O.

On the other hand, if it is determined that there is reference information in which the “multiple speaker period” flag or the “noisy period” flag is on (step S111K: Yes), the object specifying unit 1921 performs the following processing: Execute (Step S111L).
In step S <b> 111 </ b> L, the object specifying unit 1921 includes the “multi-speaker period” flag and the “noisy period” flag of the reference information immediately before the reference information among the reference information stored in the memory unit 16 in step S <b> 111 </ b> L. It is determined whether or not is in an off state. That is, the object specifying unit 1921 determines whether or not the speaker is specified in the period immediately before the reference information (both the direction and gender of the speaker can be specified). .
If it is determined that the speaker has not been specified in the immediately preceding period (step S111L: No), the electronic device 1 proceeds to step S111O.

On the other hand, when it is determined that the speaker has been specified in the immediately preceding period (step S111L: Yes), the object specifying unit 1921 sets the “multiple speaker period” flag or the “no bother period” flag to be in the on state. The speaker of the reference information is estimated as the speaker (speaker direction and gender) specified in the immediately preceding period (step S111M).
Subsequently, the reference information generation unit 193 sets the speaker of the reference information in which the “multiple speaker period” flag or the “noisy period” flag is on as the speaker estimated in step S111M, and uses the reference information as the speaker. Update (step S111N).

After step S111N, if it is determined in step S111K that there is no reference information in which the “multi-speaker period” flag or the “noisy period” flag is on (step S111K: No), or immediately before in step S111L When it is determined that the speaker has not been identified during the period (step S111L: No), the reference information generation unit 193 executes the following processing (step S111O).
In step S111O, the reference information generation unit 193 stores the reference information for each period stored in the memory unit 16 and generated in steps S111F, S111H, and S111I (or the updated reference information when updated in step S111N). Then, it is recorded in the recording unit 17 in association with the first and second audio data as the target of the speaker display processing.

After step S111O or when it is determined in step S111A that reference information has been generated (step S111A: Yes), the playback screen generation unit 194 executes the following processing (step S111P).
In step S111P, the reproduction screen generation unit 194 uses the reference information associated with the first and second audio data to be subjected to the speaker display process among the first and second audio data recorded in the recording unit 17. Based on this, a speaker display reproduction screen is generated.
Subsequently, the display control unit 195 causes the display unit 14 to display the speaker display reproduction screen generated in step S111P (step S111Q). Thereafter, the electronic device 1 returns to the main routine shown in FIG.

[Specific examples of reference information]
Next, a specific example of the reference information generated in the speaker display process (step S111) described above will be described.
FIG. 4 is a diagram illustrating an example of a situation where the first and second audio data to be subjected to speaker display processing (step S111) are generated (recorded). FIG. 5 is a diagram illustrating an example of reference information generated when the speaker display process (step S111) is executed on the first and second audio data generated in the situation of FIG.
Specifically, FIG. 4 shows a situation where three persons, a male M and a female L1, L2, make a meeting surrounding the table and record the meeting with the electronic device 1 placed on the table. . Here, when the axis Ax obtained by extending the center line of the electronic device 1 from the upper end of the electronic device 1 is used as a reference, the male M is “right (when the electronic device 1 is viewed from the front ( Assume that the person is sitting in the direction of 120 ° to the right (viewed from the upper side in FIG. 4). The woman L1 is sitting in the direction of 90 ° to the right with respect to the axis Ax. Furthermore, it is assumed that the female L2 is sitting in the direction of “10 ° to the left” with respect to the axis Ax.
In FIG. 5, one period for reading the first and second data elements in step S111B is set to 5 seconds. Therefore, in the following, each period of “0 to 5 seconds”, “5 to 10 seconds”, “10 to 15 seconds”, “15 to 20 seconds”, and “20 to 25 seconds” will be described in order.

[0-5 seconds duration]
During this period, only male M uttered. That is, during the period, the volume of the sound included in the second data element (the sound from the right side with respect to the axis Ax) is the volume of the sound included in the first data element (the sound from the left side with respect to the axis Ax). Is bigger than. In addition, since the voice is male M, the voice has a relatively low frequency. For this reason, in step S111C, the speaker is specified to be in the direction of “120 ° to the right” by the balance of the volume of each voice included in the first and second data elements in the period. Further, since the voice included in the first and second data elements has a relatively low frequency, the speaker is identified as “male”.

In addition, this period is the first period and there is no immediately preceding period. For this reason, in step S111E, it is determined that the speaker's tension is “normal”.
In step S111F, as reference information for the period, as shown in FIG. 5, the specified speaker (“male” in the direction of “120 ° to the right”) and the determined speaker's tension (“ Normal ”), a time stamp corresponding to the period (“ 9/15 11:21:10 ”), and the number of voices (“ 1 ”) are associated with each other.

[5-10 seconds period]
During this period, male M and female L1 each uttered a voice. In step S111C, the balance of the volume of each voice included in the first and second data elements in the period and the frequency of the voice (male voice has a relatively low frequency and female voice has a relatively low frequency. High) identifies the first speaker as “male” in the direction of “120 ° to the right” and the second speaker as “female” in the direction of “90 ° to the right”.
In this period, the male M speaks louder than the period “0 to 5 seconds” immediately before the period. Therefore, in step S111E, the volume of the voice of the first speaker (“male” in the direction of “120 ° to the right”) is compared with the volume of the voice of the speaker in the immediately preceding period. It is recognized that this is the case, and the tension of the speaker is determined as “high tension”. For the second speaker (“female” in the direction of “90 ° to the right”), since the speaker is not specified in the immediately preceding “0 to 5 seconds” period, in step S111E, The speaker's tension is determined as “normal”.

  In step S111F, as the reference information of the period, as shown in FIG. 5, the identified first speaker (“male” in the direction of “120 ° to the right”) and the identified speaker are identified. The tension (“high tension”), the identified second speaker (“female” in the direction of “90 ° to the right”), the determined tension of the speaker (“normal”), and the period Reference information in which a corresponding time stamp (“9/15 11:21:15”) is associated with the number of voices (“2”) is generated.

[10-15 seconds period]
During this period, male M and female L1 each uttered a voice. In the example shown in FIG. 5, in step S111C, it can be specified that the first speaker is “male” and the second speaker is “female”. This illustrates that the directions of the first speaker and the second speaker could not be specified. That is, it is not possible to specify a speaker (specification of both speaker direction and gender), but it is specified that there are a plurality (two) of speakers (step S111G: Yes). For this reason, in step S111H, as the reference information for the period, the identified first speaker (“male”) and second speaker (“female”), and a time stamp (“9 / 15 11:21:20 ") and the number of voices (" 2 ") are associated with each other, and reference information in which the" multi-speaker period "flag is turned on is generated.

Here, the speaker is specified in the period of “5 to 10 seconds” immediately before the period. For this reason, in step S111M, the identified first speaker (“male”) is estimated to be the same-sex speaker identified in the immediately preceding period (“male” in the direction of “120 ° to the right”). The Similarly, the identified second speaker (“female”) is estimated to be the same-sex speaker (“female” in the direction of “90 ° to the right”) identified in the immediately preceding period.
In step S111N, the reference information generated in step S111H includes the first speaker (“male” in the direction of “120 ° to the right”) and the tension (“ ”), The second speaker (“ female ”in the direction of“ 90 ° to the right ”), the tension of the speaker (“ normal ”), and the timestamp (“ 9/15 11:21:20 ” ) And the number of voices (“2”) are associated with each other, and the “multi-speaker period” flag is updated to reference information that is turned on. When the “multiple speaker period” flag is on, the speaker's tension is set to “normal” when the reference information is updated. The same applies to the case where the “noisy period” flag is on.

[15-20 seconds period]
During this period, female L1 and female L2 each uttered voices. Then, in step S111C, the first speaker is determined based on the balance of the volume of each voice included in the first and second data elements in the period and the frequency of the voice (difference between the voice frequencies of females L1 and L2). It is identified as “female” in the direction of “90 ° to the right” and the second speaker is “female” in the direction of “10 ° to the left”.
In addition, in the period, since the speaker is not specified in the immediately preceding “10 to 15 seconds” period, in step S111E, the first speaker (“female” in the direction of “90 ° to the right”) and Each tension of the second speaker (“female” in the direction of “10 ° to the left”) is determined as “normal”.

  In step S111F, as the reference information for the period, as shown in FIG. 5, the identified first speaker (“female” in the direction of “90 ° to the right”) and the identified speaker are identified. Tension (“normal”), the identified second speaker (“female” in the direction of “10 ° to the left”) and the determined speaker's tension (“normal”), corresponding to the period The reference information in which the time stamp (“9/15 11:21:25”) to be associated with the number of voices (“2”) is generated is generated.

[20-25 seconds period]
During this period, only woman L2 uttered. Then, in step S111C, the speaker is “10 to the left by the balance of the volume of each voice and the frequency of the voice (the female voice has a relatively high frequency) included in the first and second data elements in the period. Identified as “female” in the direction of “°”.
Moreover, in the said period, the woman L2 is louder than the period of "15-20 seconds" immediately before the said period. For this reason, in step S111E, the volume of the voice of the speaker (“female” in the direction of “10 ° to the left”) is equal to or higher than the first threshold value compared to the volume of the voice of the speaker in the immediately preceding period. It is recognized that the speaker's tension is “high tension”.

  In step S111F, as reference information for the period, as shown in FIG. 5, the identified speaker (“female” in the direction of “10 ° to the left”) and the determined tension ( Reference information in which “high tension”), a time stamp corresponding to the period (“9/15 11:21:30”), and the number of voices (“1”) are associated is generated.

[Specific example of speaker display playback screen]
Next, a specific example of the speaker display reproduction screen generated by the speaker display process (step S111) described above will be described.
FIG. 6 is a diagram showing an example of the speaker display reproduction screen W100 generated based on the reference information shown in FIG.
If the reference information recorded in the recording unit 17 in step S111O is the reference information shown in FIG. 5, a speaker display reproduction screen W100 shown in FIG. 6 is generated in step S111P.
As shown in FIG. 5, the speaker display reproduction screen W100 is a screen on which a time bar TB and first to third identification images I1 to I3 are arranged.

As shown in FIG. 6, the time bar TB is provided on the time scale SC corresponding to the time from the start to the end of voice recording and the time scale SC, and at the time of voice playback (steps S113 to S116). ) Audio data time stamp and a slider SL indicating a reproduction position corresponding in time.
The first to third identification images I1 to I3 are identification images for identifying the speaker and the tension of the speaker. In the reference information shown in FIG. 5, there are three speakers (“male” in the direction of “120 ° to the right”, “female” in the direction of “90 ° to the right”, and the direction of “10 ° to the left”. In the speaker display reproduction screen W100, three first to third identification images I1 to I3 are arranged.

The first identification image I1 is an identification image corresponding to “male” in the direction of “120 ° to the right”, which is the first speaker. In the reference information shown in FIG. 5, the speaker is specified continuously for the periods “0 to 5 seconds”, “5 to 10 seconds”, and “10 to 15 seconds”. Therefore, the first identification image I1 is arranged so as to extend along the time bar TB during the “0 to 15 seconds” period corresponding to the time bar TB.
Here, in the reference information shown in FIG. 5, the speaker is identified as “male”. Therefore, as shown in FIG. 6, a male image MF for identifying the “male” is added to the first identification image I1.
Further, in the reference information shown in FIG. 5, it is determined that the speaker's tension is “high tension” in a period of “5 to 10 seconds”. For this reason, as shown in FIG. 6, the first identification image I1 has a width dimension larger than that in other periods only during the period. That is, the width of the first identification image I1 indicates the height of the speaker's tension. The same applies to other identification images. Depending on the height of the speaker's tension, the width of the first identification image I1 may be changed in an analog manner or may be changed in stages. Moreover, you may restrict | limit so that the said width | variety may not overlap with adjacent time bar TB. In addition, as long as the aesthetic appearance is not impaired, the expression may be such that the width overlaps with the adjacent time bar TB. In this case, it goes without saying that there is a sense of realism. Further, the width of the first identification image I1 may be constant, and the height of the tension may be represented by the size of the added male image MF.

The second identification image I2 is an identification image corresponding to “female” in the direction of “90 ° to the right”, which is the second speaker. In the reference information shown in FIG. 5, the speaker is specified continuously for the periods of “5 to 10 seconds”, “10 to 15 seconds”, and “15 to 20 seconds”. For this reason, the second identification image I2 is arranged so as to extend along the time bar TB during the “5 to 20 seconds” period corresponding to the time bar TB.
Here, in the reference information shown in FIG. 5, the speaker is identified as “female”. For this reason, as shown in FIG. 6, a female image LF1 for identifying the “female” is added to the second identification image I2.
Further, in the reference information shown in FIG. 5, it is determined that all the tensions of the speaker are “normal”. For this reason, as shown in FIG. 6, the second identification image I2 has the same width dimension in all periods.

The third identification image I3 is an identification image corresponding to “female” in the direction of “10 ° to the left”, which is the third speaker. In the reference information shown in FIG. 5, the speaker is specified continuously for the period of “15 to 20 seconds” and “20 to 25 seconds”. Therefore, the third identification image I3 is arranged so as to extend along the time bar TB during the “15 to 25 seconds” period corresponding to the time bar TB.
Here, the reference information shown in FIG. 5 specifies that the speaker is “female” different from the second speaker (“female” in the direction of “90 ° to the right”). For this reason, as shown in FIG. 6, the third identification image I3 is added with a female image LF2 that is an image for identifying the “female” and is different from the female image LF1.
Further, in the reference information shown in FIG. 5, the tension of the speaker is determined to be “high tension” in a period of “20 to 25 seconds”. For this reason, as shown in FIG. 6, the third identification image I3 has a width dimension larger than that in the other periods only during the period.

The electronic device 1 according to the first embodiment described above analyzes the first and second sound data to determine the sound feature component (speaker tension) included in the sound data, and the feature component. And the time (time stamp) in which the characteristic component is included are associated with each other to generate reference information. Based on the reference information, the electronic device 1 reproduces speaker display in which an identification image for identifying a characteristic component associated with the time is arranged at each time corresponding to the time bar together with the time bar. Display the screen.
In particular, the electronic device 1 uses different identification images depending on the speaker's tension (in the example shown in FIG. 6, the speaker's tension is expressed by the width of the identification images I1 to I3).
For this reason, the user can grasp at a glance the situation at the time of recording (how the tension of the speaker was) from the speaker display reproduction screen. Therefore, according to the electronic device 1 according to the first embodiment, the convenience can be improved.

In addition, the electronic apparatus 1 according to the first embodiment analyzes the first and second sound data, identifies the speaker that has emitted the sound included in the first and second sound data, and identifies the speaker For each speaker, feature information (speaker tension) is determined.
For this reason, the user can grasp at a glance both who is the speaker and the state of the speaker's tension as a recording situation from the speaker display reproduction screen. Further improvement in performance can be achieved.
In particular, the electronic device 1 is provided with a pair of first and second microphones 111 and 121, and is based on first and second sound data based on each sound input via the first and second microphones 111 and 121. Identify the direction of the speaker. In addition, the electronic device 1 identifies the sex of the speaker based on the frequency of each voice. Furthermore, the electronic device 1 determines the speaker's tension based on the volume of each voice. Therefore, speaker identification (speaker direction and gender identification) and speaker tension determination can be performed by simple analysis processing.

Also, in the case where the electronic device 1 according to the first embodiment cannot identify the speaker, the electronic device 1 transmits the voice of the period in which the speaker cannot be identified in the period immediately before the period. Presume that the identified speaker originated.
For this reason, even if it is a case where a speaker cannot be specified, it can estimate that it is uttering continuously in each period, and can estimate the said speaker as a likely speaker.

(Modification of Embodiment 1)
In the first embodiment described above, when the electronic device 1 is set to the playback mode (step S111), the analysis of the first and second audio data and the generation of the reference information are performed. Absent.
For example, at least part of the analysis of the first and second sound data and the generation of the reference information (steps S111B to S111N) is performed in parallel with the generation of the first and second sound data (steps S103 to S105). It doesn't matter.

  In the first embodiment described above, when the first and second data elements are analyzed and a laughing voice or an angry voice can be recognized, the period during which the recognition can be performed on the speaker display reproduction screen An image of a laughing face or an angry face may be added at a position corresponding to.

  In the first embodiment described above, the tension of the speaker is determined in two stages of “high tension” and “normal”. However, the present invention is not limited to this and may be determined in three or more stages.

  In the first embodiment described above, when determining the tension of the speaker, the volume of the speaker's voice is compared between the corresponding period and the immediately preceding period. However, the present invention is not limited to this. The speaker's tension may be determined by comparing the volume of the speaker with a predetermined threshold. Further, the tension of the speaker may be determined based on a change in the volume of the speaker's voice within the corresponding period. Here, the speaker's tension (speech feature) is assumed to be emotionally high, but the concentration of the story (for example, one speaker explains and others quietly listen to it) Etc.) may be reflected. In this case, the speaker's tension is detected by detecting the relative loudness of the detected voices of multiple people, the degree of the pace of the words (speaking to speak), the speed (speaking), etc. May be determined. In other words, the speaker's tension can detect relative changes (such as starting to concentrate) over the course of a single speaker's voice, or detect the relative differences between multiple people's voices. Is determined.

  In the first embodiment described above, if the speaker cannot be specified, the speaker in which the voice during the period when the speaker could not be specified is specified in the “immediately before” period of the period. (Step S111M) is not limited to this, but it may be estimated that the speaker specified in the “immediately after” period is the speaker.

FIG. 7 is a diagram showing a modification of the first embodiment of the present invention.
Instead of the electronic device 1 described in the first embodiment, an electronic device 1A to which an imaging function is added may be employed.
Specifically, as illustrated in FIG. 7, the electronic device 1 </ b> A has an imaging unit 10 added to the electronic device 1 described in the first embodiment, and instead of the device-side control unit 19. A device-side control unit 19A in which an imaging control unit 197 is added to the device-side control unit 19 is employed.
The imaging unit 10 captures a subject and generates image data under the control of the imaging control unit 197. The imaging unit 10 includes an optical system (not shown) that forms a subject image, an imaging element such as a CCD (Charge Coupled Device) that receives the subject image formed by the optical system and converts it into an electrical signal, and the imaging It is configured using a signal processing unit that generates digital image data by performing signal processing (A / D conversion or the like) on an electrical signal (analog signal) from the element. The image data generated by the imaging unit 10 is added with a time stamp (time stamp related to the date and time when the image data was generated) generated by the clock unit 15 under the control of the imaging control unit 197. Are recorded in the recording unit 17.
The imaging control unit 197 causes the imaging unit 10 to image the subject in response to a shooting operation on the operation unit 13 by the user, and the image data (including a time stamp) generated by the imaging unit 10 is stored in the recording unit 17. Record.
As described above, if the electronic device 1A is provided with an imaging function, for example, the situation of the meeting shown in FIG. 4 and the images of the male M and the females L1 and L2, respectively, for example, the speaker shown in FIG. On the display reproduction screen W100, it is possible to arrange an image obtained by imaging the meeting situation shown in FIG. 4 and an image obtained by imaging the male M and the female L1, L2 instead of the male image MF and the female images LF1 and LF2. .

(Embodiment 2)
Next, a second embodiment of the present invention will be described.
In the following description, the same reference numerals are given to the same configurations and steps as those in the above-described first embodiment, and the detailed description thereof is omitted or simplified.
FIG. 8 is a block diagram showing the configuration of the speech processing system 100 according to Embodiment 2 of the present invention.
As shown in FIG. 8, the voice processing system 100 according to the second embodiment provides the server 2 with the function of “analyzing voice data and generating reference information” of the electronic device 1 described in the first embodiment. This is a system that performs communication via the Internet network N between the electronic device 1B that generates and reproduces audio data and the server 2.

[Configuration of voice processing system]
Hereinafter, the configuration of the electronic device 1B and the server 2 constituting the voice processing system 100 according to the second embodiment will be described in order.

[Configuration of electronic equipment]
As shown in FIG. 8, the electronic device 1B according to the second embodiment has a device-side communication unit 20 added to the electronic device 1 (FIG. 1) described in the first embodiment. Some functions of the device-side control unit 19 are changed.
The device side communication unit 20 is a communication interface for performing wireless communication of various data including signals necessary for communication with the server 2 under the control of the device side control unit 19B.

As shown in FIG. 8, in the device-side control unit 19B according to the second embodiment, the audio data analysis unit 192 and the reference information generation unit 193 are omitted, and the device-side communication control unit 198 is added. .
The device-side communication control unit 198 executes the following process when the speaker display process is executed.
Specifically, the device-side communication control unit 198 connects to the Internet network N via the device-side communication unit 20 based on the location information (URL (Uniform Resource Locator)) of the server 2 recorded in the recording unit 17. An access signal (a request for transmitting a reference image (including identification information for identifying its own electronic device 1B)) is transmitted to the connected server 2 to establish a communication connection with the server 2. And the apparatus side communication control part 198 transmits the 1st, 2nd audio | voice data (a time stamp is included) used as the object of a speaker display process to the server 2, and receives reference information from the server 2. FIG.

[Server configuration]
The server 2 analyzes the first and second audio data transmitted together with the reference image transmission request from the electronic device 1B, generates reference information, and transmits the reference information to the electronic device 1B.
In the following, the main part of the present invention will be mainly described as the configuration of the server 2.
As shown in FIG. 8, the server 2 includes a server-side communication unit 21, a voice database 22, and a server-side control unit 23.

The server-side communication unit 21 is a communication interface for performing wireless communication of various data including signals necessary for communication with the electronic device 1B under the control of the server-side control unit 23.
The voice database 22 records first and second voice data (including a time stamp) received from the electronic device 1B via the server-side communication unit 21 under the control of the server-side control unit 23. The voice database 22 records reference information in association with the first and second voice data used for generating the reference information under the control of the server-side control unit 23.

The server-side control unit 23 is configured using a CPU or the like, and comprehensively controls the operation of the server 2. As shown in FIG. 8, the server-side control unit 23 includes a server-side communication control unit 231, a terminal determination unit 232, an audio data recording control unit 233, an audio data analysis unit 234, and a reference information generation unit 235. Is provided.
The server-side communication control unit 231 responds to an access signal (reference information transmission request (including identification information of the electronic device 1B)) transmitted from the electronic device 1B via the server-side communication unit 21 and the Internet network N. Then, the operation of the server side communication unit 21 is controlled to establish a communication connection with the electronic device 1B. Then, the server-side communication control unit 231 receives the first and second audio data (including time stamp) to be recorded in the audio database 233 from the electronic device 1B, and the reference generated by the reference information generation unit 235. Information is transmitted to the electronic device 1B.
And the server side communication control part 231 has a function as an audio | voice data acquisition part which concerns on this invention.

Based on an access signal transmitted from the electronic device 1B via the Internet N, the terminal determination unit 232 determines (identifies) the transmission-source electronic device 1B that has accessed.
The audio data recording control unit 233 records the first and second audio data (including the time stamp) received from the electronic device 1B via the server side communication unit 21 in the audio database 22.

The sound data analysis unit 234 (the object specifying unit 2341 and the feature component determining unit 2342) is the same as the sound data analyzing unit 192 (the object specifying unit 1921 and the feature component determining unit 1922) described in the first embodiment. It has a function and analyzes the first and second sound data received from the electronic device 1B via the server side communication unit 21 and recorded in the sound database 22.
The reference information generation unit 235 has the same function as the reference information generation unit 193 described in the first embodiment, and generates reference information based on the analysis result of the audio data analysis unit 234. Then, the reference information generation unit 235 records the reference information in the audio database 22 in association with the first and second audio data used for generating the reference information.
And the server side communication control part 231, the audio | voice data analysis part 234, and the reference information generation part 235 have a function as an audio | voice processing apparatus which concerns on this invention.

[Operation of voice processing system]
Next, the operation of the voice processing system 100 described above will be described.
Hereinafter, as operations of the voice processing system 100, operations of the electronic device 1B and operations of the server 2 will be described in order.

[Operation of electronic equipment]
The operation of electronic device 1B is different from the operation of electronic device 1B described in the first embodiment (FIGS. 2 and 3) only in the speaker display process (step S111). Therefore, hereinafter, only the speaker display process (step S111) according to the second embodiment will be described.
FIG. 9 is a flowchart showing speaker display processing (step S111) according to Embodiment 2 of the present invention.
In the speaker display process according to the second embodiment, as shown in FIG. 9, steps S111A to S111O are omitted from the speaker display process (FIG. 3) described in the first embodiment. The only difference is the addition of S111R and S111S. Therefore, only steps S111R and S111S will be described below.

Step S111R is a step executed at the beginning of the speaker display process (step S111).
Specifically, the device-side communication control unit 198 sends an access signal (reference image transmission request (including identification information of its own electronic device 1B) to the server 2 connected to the Internet network N via the device-side communication unit 20. )) To establish a communication connection with the server 2. Then, the device-side communication control unit 198 sends the first and second voice data (first and second voice data (including time stamp) selected in step S109) to be the target of the speaker display process to the server 2. Send.

Subsequently, the device-side communication control unit 198 receives reference information from the server 2 via the device-side communication unit 20 and stores it in the memory unit 16 (step S111S).
Then, the electronic device 1B generates a speaker display reproduction screen based on the reference information stored in the memory unit 16 (step S111P), and displays the speaker display reproduction screen on the display unit 14 (step S111Q).

[Server operation]
FIG. 10 is a flowchart showing the operation of the server 2.
Whether the server-side communication control unit 231 has received an access signal (a reference image transmission request (including identification information of the electronic device 1B)) from the electronic device 1B via the server-side communication unit 21 and the Internet network N. Is determined (step S201).
If it is determined that the reference image transmission request has not been received (step S201: No), the server 3 proceeds to step S210.
On the other hand, when it is determined that the reference image transmission request has been received (step S201: Yes), the terminal determination unit 232 identifies the electronic device 1B that has accessed, based on the transmission request ( Step S202).

  Subsequently, the server side communication control unit 231 receives the first and second audio data (including the time stamp) from the electronic device 1B via the server side communication unit 21 and the Internet network N (step S203: audio data). Acquisition step). Then, the audio data recording control unit 233 refers to the first and second audio data recorded in the audio database 22, and the same first and second audio data as the first and second audio data received in step S203. Is not recorded yet, the received first and second audio data are recorded in the audio database 22.

Subsequently, the server-side control unit 23 determines whether reference information for the first and second audio data received in step S203 has already been generated (step S204). That is, the server-side control unit 23 determines whether or not reference information is associated with the first and second audio data recorded in the audio database 22 in step S204.
When it is determined that the reference information has been generated (step S204: No), the server 2 proceeds to step S219.
On the other hand, when it is determined that the reference information has not yet been generated (step S204: Yes), the server 2 receives the information in step S203 as in steps S111B to S111N described in the first embodiment. Analysis of the first and second audio data and generation of reference information are executed (steps S205 to S217).
That is, step S208 corresponds to the audio data analysis step according to the present invention. Step S209 corresponds to a reference information generation step according to the present invention.

After step S217 or when it is determined in step S215 that the speaker has not been specified in the immediately preceding period (step S215: No), the server 2 generates each of the steps generated in steps S209, S211 and S212. The reference information of the period (or the updated reference information when updated in step S217) is recorded in the voice database 22 in association with the first and second voice data used for generating the reference information (step S218). ).
After step S218 or when it is determined that the reference information has been generated in step S204 (step S204: No), the server side communication control unit 231 passes through the server side communication unit 21 and the Internet network N. The reference information recorded in association with the first and second audio data received in step S203 in step S218 is transmitted to the electronic device 1B specified in step S202 (step S219). Thereafter, the server 2 returns to step S201.

  If it is determined in step S201 that a reference image transmission request has not been received (step S201: No), the server 2 executes another process different from the process described above (step S210). Thereafter, the server 2 returns to step S201.

  According to the second embodiment described above, the same effects as those of the first embodiment described above can be achieved, and the configuration of the electronic device 1B can be simplified.

(Other embodiments)
The embodiments for carrying out the present invention have been described so far, but the present invention should not be limited only by the above-described first and second embodiments.
11A to 11C are diagrams showing modifications of the speaker display reproduction screen described in the first and second embodiments.
In the speaker display reproduction screen W100 exemplified in the first and second embodiments described above, the first to third identification images I1 to I3 express the speaker's tension with the thickness of the width. Instead, for example, the speaker display reproduction screens W101 to W103 shown in FIGS. 11A to 11C may be expressed.
Specifically, on the speaker display reproduction screen W101 shown in FIG. 11A, the first to third identification images I1 to I3 express changes in the speaker's tension as waveforms. That is, in the speaker display reproduction screen W101 shown in FIG. 11A, the vertical direction indicates the height of the tension.
A speaker display reproduction screen W102 shown in FIG. 11B is a 3D display of the speaker display reproduction screen W101 shown in FIG. 11A.
Further, on the speaker display reproduction screen W103 shown in FIG. 11C, the first to third identification images I1 to I3 express the speaker's tension as a pixel value. That is, in the speaker display reproduction screen W101 shown in FIG. 11C, a portion with a high pixel value (bright portion) indicates a time during which the tension is high. Here, the speaker's tension shows an emotional height, but even if it reflects the concentration of the story (for example, one speaker explains it and others listen quietly) Good. In this case, the speaker's tension is detected by detecting the relative loudness of the voices of multiple people detected, the degree of the pace of the words (speaking to speak) and the speed (speaking). May be determined. For example, it is possible to perform a search for determining the situation based on the characteristics (changes) of the voice.

FIG. 12 is a diagram showing a modification of the reference information described in the first and second embodiments.
In the first and second embodiments, the reference information is not limited to the reference information (for example, FIG. 5) described in the first and second embodiments. For example, the reference information illustrated in FIG. I do not care.
For example, in Embodiment 1 described above, a specific keyword is recorded in advance in the recording unit 17. In FIG. 12, for convenience of explanation, only one specific keyword is used, but a plurality of specific keywords may be used. Further, the voice data analysis unit 192 analyzes the first and second data elements and determines whether or not the specific keyword recorded in the recording unit 17 is included in the first and second data elements. When the voice data analysis unit 192 determines that the specific keyword is included, the reference information generation unit 193 has a corresponding period (a period of “5 to 10 seconds” in the example of FIG. 12). As reference information, reference information with the “keyword” flag turned on is generated. In the second embodiment described above, the server 2 may execute the above-described processing to generate the reference information illustrated in FIG.

FIG. 13 is a diagram showing an example of the speaker display reproduction screen W104 generated based on the reference information shown in FIG.
For example, in the first and second embodiments described above, when the electronic device 1 or the server 2 generates the reference information shown in FIG. 12, the electronic devices 1 and 1B, for example, reproduce the speaker display shown in FIG. A screen W104 is generated.
Specifically, in the speaker display reproduction screen W104 shown in FIG. 13, a keyword input unit KW is added to the speaker display reproduction screen W100 shown in FIG.
The keyword input unit KW is a part where a keyword is input by an operation on the operation unit 13 by a user.
Then, when the same keyword as the specific keyword recorded in the recording unit 17 is input by the user operating the operation unit 13, the playback screen generation unit 194 performs the input before the input (FIG. 13A). ) And after the input (FIG. 13B), the speaker display reproduction screen W104 is changed as shown below.
That is, the playback screen generation unit 194 refers to the reference information shown in FIG. 12, and identifies the identification image corresponding to the speaker during the period in which the “keyword” flag is on (FIGS. 12 and 13B). In the example, a speaker display reproduction screen W104 (FIG. 13B) in which the luminance of the second identification image I2) is improved is generated.
In addition, if the display mode of the identification image corresponding to the speaker in the period in which the “keyword” flag is on is different from the display mode of the identification image, the brightness is not limited to the above-described improvement. Other methods may be adopted.

The processing flow is not limited to the processing order in the flowcharts described in the first and second embodiments, and may be changed within a consistent range.
Furthermore, the processing algorithm described using the flowcharts in this specification can be described as a program. Such a program may be recorded on a recording unit inside the computer, or may be recorded on a computer-readable recording medium. Recording of the program in the recording unit or recording medium may be performed when the computer or recording medium is shipped as a product, or may be performed by downloading via a communication network.

  In the above-described embodiment, the description has been made by taking the search technology such as an IC recorder and a recorder as an example for easy understanding. However, in the case of a video shooting system linked with a video camera or the like, audio and images are associated with each other. It can also be applied to cameras with recording functions. It is possible to record, search, and tag captured images based on speaker tension.

  In addition to general cameras, in-vehicle cameras can also be used for taking pictures and searching for images through in-car conversations, so it is possible to determine the driver and perform hands-free photography while driving. . In addition, surveillance cameras, inspection cameras, and medical cameras can search for images related to a conversation of a specific person. Even when applied to an inspection apparatus, not only the image to be inspected but also the scenery being inspected can check the important scene based on the conversation, and the effect unique to the present invention can be expected.

  DESCRIPTION OF SYMBOLS 1, 1A, 1B ... Electronic device; 2 ... Server; 10 ... Imaging part; 11 ... 1st audio | voice data generation part; 12 ... 2nd audio | voice data generation part; Operation unit; 14 ... display unit; 15 ... clock unit; 16 ... memory unit; 17 ... recording unit; 18 ... audio output unit; 19, 19A, 19B ... device side control 20: device side communication unit; 21 ... server side communication unit; 22 ... voice database; 23 ... server side communication control unit; 100 ... voice processing system; 1st microphone; 112 ... 1st amplification part; 113 ... 1st A / D conversion part; 121 ... 2nd microphone; 122 ... 2nd amplification part; 123 ... 2nd A / D conversion part 181 ... D / A converter; 182 ... Amplifier; 183 ... Speaker; 191 ... Audio data Obtaining unit: 192: Audio data analyzing unit; 193: Reference information generating unit; 194: Reproduction screen generating unit; 195 ... Display control unit: 196 ... Audio control unit: 197 ... Imaging control unit: 198: Device side communication control unit; 231: Server side communication control unit; 232: Terminal determination unit; 233: Audio data recording control unit: 234: Audio data analysis unit 235 ... Reference information generation unit; 1921 ... Object identification unit; 1922 ... Feature component discrimination unit; 2341 ... Object identification unit; 2342 ... Feature component discrimination unit; Ax ... Axis; I1-I3 ... 1st-3rd identification image; KW ... Keyword input part; L1, L2 ... Female; LF1, LF2 ... Female image; M ... Male;・ Male image; N ... Internet network; SC ... Time scan Lumpur; SL ··· slider; TB ··· time bar; W100~W104 ··· speaker display playback screen

Claims (10)

An audio data acquisition unit for acquiring audio data;
An audio data analysis unit that analyzes the audio data and discriminates a high tension component of the audio included in the audio data;
A reference information generation unit that associates the high tension component and the time during which the high tension component is included in the audio data, and generates reference information used when generating the reproduction screen of the audio data ;
The voice data analysis unit
Analyzing the voice data, for each predetermined time range in the voice data, identifying the speaker that has produced the voice contained in the voice data and determining the high tension component, and identifying the speaker If it is not possible to do, and characterized in that the voice of the speaker time range that can not be identified, you estimated that shortly before or speaker identified in the time range immediately uttered for that time range Voice processing device. The voice data analysis unit
2. The voice data is analyzed to identify a speaker who has produced a voice included in the voice data, and the presence or absence of the high tension component is determined for each of the identified speakers. The voice processing apparatus according to 1. The voice data analysis unit
Speaker's emotional height, relative loudness of multiple people's voices, constant pace of words spoken by the speaker, speaker speed, voice volume, voice frequency, phoneme component of voice The speech processing apparatus according to claim 1, wherein the tension of the speaker is determined by detecting the component having a high tension based on the time density of the voice or a specific voice component. And a reproduction screen generation unit that generates a speaker display reproduction screen in which an identification image for identifying a tension associated with the time is arranged based on the reference information generated by the reference information generation unit. The speech processing apparatus according to any one of claims 1 to 3. The playback screen generation unit
The voice processing apparatus according to claim 4, wherein an icon to be displayed for identifying a speaker is generated in the identification image for identifying the tension. The identification image is
The voice included in the voice data is displayed so as to correspond to the time, and the period indicating the speaker's tension is generated as data that is displayed wider than other periods. The speech processing apparatus according to claim 5. The identification image is
The voice included in the voice data is displayed so as to correspond to the time, and the period indicating the speaker's tension is generated as data that is displayed in a stepwise manner that is wider in analog than the other periods. The speech processing apparatus according to claim 5, wherein: It has an operation reception unit that accepts keyword input operations,
The playback screen generation unit
When the specific sound based on the high tension component matches the keyword received by the operation reception unit, the identification image for identifying the high tension component is different from the previous identification image. The speech processing apparatus according to claim 4 or 5, wherein: In the speech processing method performed by the speech processing apparatus,
An audio data acquisition step for acquiring audio data;
An audio data analysis step of analyzing the audio data and discriminating a high tension component of the audio included in the audio data;
Associating the time and included high component of the tension in the high component and the audio data of the tension, see contains a reference information generating step of generating a reference information used in generating the playback screen of the voice data,
In the voice data analysis step,
Analyzing the voice data, for each predetermined time range in the voice data, identifying the speaker that has produced the voice contained in the voice data and determining the high tension component, and identifying the speaker When it is not possible to do so, it is presumed that the voice in the time range in which the speaker cannot be specified is emitted by the speaker specified in the time range immediately before or immediately after the time range. Audio processing method. A speech processing program for causing a speech processing device to execute the speech processing method according to claim 9 .