JP6193826B2 - Pulsation sensation information generation apparatus, pulsation sensation information generation method, distribution system, and program - Google Patents

Description

  The present invention relates to a technique for presenting a sensation that an animal feels for its own heartbeat (hereinafter referred to as “beating sensation”).

  As a method of sharing the performer's psychological state and physical state with the audience, a technique for distributing biological information itself such as heartbeat and electrocardiogram acquired from the performer to the audience is known (see Patent Document 1).

JP 2005-303722 A

  Humans can feel that their heart is beating ("beating sensation"). The pulsation sensation is a sensation that an animal feels for the beat of its own heart, and is a perceptual phenomenon in which physical changes caused by the pulsation of the heart in the animal's own body are grasped through the sensory organs. For example, in a quiet room, if you focus on the heart beat in a calm state, you will feel a long beat of “Dokun… Dokun…”, and after an intense exercise, a short beat of “Dokudukudoku…” Feel the movement. In order to present and share with others the beating sensation felt by a person, the time interval (hereinafter also referred to as “beating interval”) and the intensity (hereinafter referred to as “beating sensation”) It is necessary to convey information about “strength”. That is, information for presenting and sharing a heartbeat feeling (hereinafter also referred to as “beating feeling information”) can be mainly expressed by a heartbeat feeling interval and a heartbeat feeling intensity. The beating feeling interval and the beating feeling intensity are those that the animal feels against the beat of its own heart and is subjective. However, the heartbeat interval is easily acquired using various biosensors as described below, but the heartbeat intensity is to be acquired unless using a dedicated measuring device or psychological experiment. Is difficult.

  In the conventional method such as Patent Document 1, the audience can share the heartbeat feeling interval (that is, the heartbeat interval) of the performer, but the audience cannot share the heartbeat feeling intensity. Therefore, it was difficult for the audience to share the psychological state such as the tension and excitement of the performers. In addition, even if the heartbeat information is expressed only by the heartbeat interval, in the conventional method, the performer must perform with the biosensor attached. For this reason, there is a problem that the biometric sensor interferes with the performer's performance.

  The present invention estimates at least one of a beating feeling interval and a beating feeling intensity from information acquired by a method that does not interfere with a subject (for example, a performer) who shares the beating feeling, and generates beating feeling information. An object of the present invention is to provide an apparatus and a method for presenting.

  In order to solve the above-described problem, according to one aspect of the present invention, a beating sensation information generation device uses a sense that an animal feels for the beat of its own heart as a beating sensation, and sets an interval between beating sensations. A beat that estimates at least one of the beating feeling interval and the beating feeling intensity from the utterance of the subject, with the interval, the strength of the beating feeling as the beating feeling intensity, the information for presenting the beating feeling as the beating feeling information A sensation information estimation unit; and a pulsation sensation information synthesis unit that synthesizes pulsation feeling information from at least one of an estimation value of an interval between pulsation feelings and an estimation value of pulsation feeling intensity.

  In order to solve the above problems, according to another aspect of the present invention, a distribution system includes a distribution server and a beating sensation information generation device. The distribution system is used to present the sensation of pulsation, with the sensation that the animal feels for the heart beat as the pulsation sensation, the interval between the pulsation sensations as the pulsation sensation interval, The information is regarded as beating feeling information, and the distribution server estimates at least one of the beating feeling interval and the beating feeling intensity using an audio signal including the utterance of the target person collected by the microphone, and estimates the beating feeling interval. At least one of the value and the estimated value of the pulsation feeling intensity is delivered to the pulsation feeling information generation device, and the pulsation feeling information generation device receives the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity received from the distribution server. A heartbeat information presenting section that combines and presents heartbeat information from at least one of them, and the heartbeat information presented by the heartbeat information presentation section includes (1) a waveform of one heartbeat prepared in advance Waveform that repeats with the estimated value of the sense interval And / or a waveform synthesized such that the amplitude of the waveform increases as the estimated value of the pulsation feeling intensity increases, and / or (2) a pre-prepared sound for one beat The sound is repeated with the estimated value of the feeling interval and / or the sound is synthesized so that the sound volume increases as the estimated value of the heartbeat feeling intensity increases, and / or (3) prepared in advance The vibration for one beat is repeated with the estimated value of the beating feeling interval, and / or the vibration is synthesized so that the larger the estimated value of the beating feeling intensity is, the larger the vibration intensity is. .

In order to solve the above problems, according to another aspect of the present invention, a distribution system includes a distribution server and a beating sensation information generation device. The distribution system is used to present the sensation of pulsation, with the sensation that the animal feels for the heart beat as the pulsation sensation, the interval between the pulsation sensations as the pulsation sensation interval, The information is regarded as heartbeat information, and the distribution server receives an audio signal including the utterance of the target person collected by the microphone and distributes it to the heartbeat information generating device. The heartbeat information generating device is picked up by the microphone. Using a speech signal including the speech of the subject, at least one of a beating feeling interval and a beating feeling intensity is estimated, and at least one of an estimated value of the beating feeling interval and an estimated value of the beating feeling intensity Including a beating sensation information presenting unit that presents the beating sensation information synthesized from any of the above, the beating sensation information presented by the beating sensation information presenting unit includes (1) a pre-prepared waveform for one heartbeat. A waveform that repeats with an estimate of the interval. , And / or a synthesized waveform so that the amplitude of the higher estimate of beating feeling intensity waveform becomes large,
And / or (2) a sound of one heart beat prepared in advance is repeated with an estimated value of the beating feeling interval, and / or the sound volume increases as the estimated value of the beating feeling intensity increases. And / or (3) vibration prepared by pre-preparing one beat of the heartbeat is repeated with an estimated value of the beating feeling interval and / or of the beating feeling intensity. The vibration is synthesized such that the greater the estimated value, the greater the vibration intensity.

  In order to solve the above-described problem, according to another aspect of the present invention, a beating sensation information generation method uses a sensation that an animal feels for the pulsation of its own heart as a pulsation sensation, and an interval between beating sensations is beaten. Estimate at least one of the heartbeat interval and the heartbeat intensity from the utterance of the subject, using the interval as the feeling interval, the strength of the heartbeat as the heartbeat intensity, and the information for presenting the heartbeat as the heartbeat information. A pulsation feeling information estimation step, and a pulsation feeling information synthesis step of synthesizing pulsation feeling information from at least one of an estimated value of the pulsation feeling interval and an estimated value of the pulsation feeling intensity.

  According to the present invention, there is an effect that it is possible to generate heartbeat information of a subject only from the speech of the subject.

The functional block diagram of the beating feeling information generation apparatus which concerns on 1st embodiment. The figure which shows the processing flow of the pulsation feeling information generation apparatus which concerns on 1st embodiment. The figure which shows the example of an image | video in the case of presenting visual information (video | video) as beating feeling information. The functional block diagram of the pulsation feeling information generation device concerning a second embodiment. The figure which shows the processing flow of the beating feeling information generation apparatus which concerns on 2nd embodiment. The functional block diagram of the pulsation feeling information generation device concerning a third embodiment. The figure which shows the processing flow of the pulsation feeling information generation apparatus which concerns on 3rd embodiment. FIG. 8A is a diagram showing an example of the relationship between the speech rate and the fundamental frequency and the heartbeat feeling interval, and FIG. 8B is a diagram showing an example of the relationship between the speech speed and the fundamental frequency and the heartbeat feeling intensity. The figure which shows the structural example of the delivery system which concerns on 4th embodiment. The figure which shows the processing flow of the delivery system which concerns on 4th embodiment. The figure which shows the usage example of the delivery system which concerns on 4th embodiment.

  Hereinafter, embodiments of the present invention will be described. In the drawings used for the following description, constituent parts having the same function and steps for performing the same process are denoted by the same reference numerals, and redundant description is omitted.

<Points of first embodiment>
In the present embodiment, by using the correlation between the heartbeat interval (beat interval) and the heartbeat intensity (beat strength) and the speed of speech, the approximate heartbeat interval and heartbeat from the speed of speech. The intensity is estimated, and beating feeling information is synthesized and presented from the estimated value of the beating feeling interval and the estimated value of the beating feeling intensity.

<First embodiment: Utilization of speech rate>
FIG. 1 is a functional block diagram of a beating sensation information generating apparatus 100 according to the first embodiment, and FIG. 2 is a processing flow thereof.

In the present embodiment, the beating feeling interval and the beating feeling intensity are estimated from the speech rate. In general, the speaking rate when a person is nervous or excited tends to be faster than the speaking rate during normal times (see Reference 1). When the person is nervous or excited, the heartbeat interval (beat interval) tends to be short and the heartbeat intensity tends to increase. Therefore, in the present embodiment, the beating feeling interval and the beating feeling intensity are estimated based on the utterance speed for each of a plurality of sections (frames).
(Reference 1) Murray, Iain R. and Arnott, John L., “Implementation and Testing of a System for Producing Emotion-by-rule in Synthetic Speech”, Speech Communication, Vol. 16, No. 4, pp.369 -390, 1995.

  The heartbeat information generating device 100 includes a microphone 101, a voice recognition device 103, an utterance speed calculation unit 105, a heartbeat interval estimation unit 110, a heartbeat feeling strength estimation unit 120, a heartbeat feeling information synthesis unit 140, and a heartbeat. A sensation information presentation unit 150. Hereinafter, processing of each unit will be described.

<Microphone 101>
The microphone 101 is attached to a subject who wants to generate heartbeat information (for example, a performer such as a stage) and collects the speech of the subject (S101). The target person's voice signal collected by the microphone 101 is input to the voice recognition device 103.

<Voice recognition device 103>
A voice signal collected by the microphone 101 is input to the voice recognition device 103 in units of frames that are predetermined time intervals. The speech recognition apparatus 103 performs phoneme labeling on the input speech signal in units of frames. As a result, a phoneme sequence corresponding to the audio signal in units of frames is generated (S103). The sequence of phonemes in units of frames output from the speech recognition apparatus 103 is input to the speech rate calculation unit 105.

As a phoneme labeling method, a conventional speech recognition technique may be used (for example, see Reference 2).
(Reference 2) Yonezawa, Mizuno, Abe, "Study on robustness of automatic labeling using HMM phoneme model", IEICE technical report. SP, Speech 102 (292), 17-22, 2002-08-23 .

<Speech rate calculator 105>
The speech rate calculation unit 105 obtains a speech rate (SR: Speech rate) from the input sequence of phonemes in units of frames (S105) and outputs it. For example, the number of uttered phonemes per second is used as the utterance speed. In this case, a value obtained by dividing the total number of phonemes included in the frame by the frame length (seconds) is output as the speech rate SR of the frame.

<Beat Feeling Interval Estimation Unit 110>
The beating sensation interval estimation unit 110 estimates the beating feeling interval (that is, the heartbeat interval) using the utterance speed SR output from the utterance speed calculation unit 105 (S110), and the estimated value is used as the beating sensation information synthesis unit 140. Output to.

  At this time, the estimated value of the beating feeling interval corresponding to the frame whose speech rate SR is the first value is more than the estimated value of the beating feeling interval corresponding to the frame whose speaking rate SR is the second value smaller than the first value. Is also estimated to be small (short).

  That is, the heartbeat interval is estimated so that the estimated value of the heartbeat interval is smaller (shorter) in a frame with a high (fast) speech rate SR than a frame with a low (slow) speech rate. For silent frames (that is, frames with SR = 0), it is assumed that the immediately preceding beating feeling interval continues, and the same value as the estimated value of the beating feeling interval of the immediately preceding non-silent frame is used. It will be applied as an estimated value.

  The following are four methods for estimating the beating feeling interval using the speech rate SR.

(Estimation method 1)
For example, in a frame in which the speech rate is equal to or less than a predetermined threshold, the beating feeling interval is estimated as a predetermined interval (eg, 1200 ms). The “predetermined interval” may be calculated from the heart rate of the subject measured in advance in a calm state, for example. In addition, in a frame in which the speech rate is greater than a predetermined threshold, the beating feeling interval is estimated to be shorter (for example, 750 ms) than the predetermined interval. As the threshold value, for example, an average of the speaking rate of the subject measured in advance in a calm state may be used.

(Estimation method 2)
Alternatively, in a frame in which the speech rate is included in a predetermined range, the beating feeling interval of the frame is estimated as a predetermined interval (eg, 1200 ms), and in a frame in which the speech rate is less than the lower limit of the predetermined range It is estimated that the heartbeat interval is longer than the predetermined interval (for example, 1400 ms), and the heartbeat interval is shorter than the predetermined interval (for example, 750 ms) in a frame in which the speech rate is larger than the upper limit of the predetermined range. May be estimated. The predetermined range is, for example, if the minimum value (or minimum value -α) and the maximum value (or maximum value + α) of the speech rate for each frame of the subject measured in a calm state in advance are the lower limit and the upper limit, respectively. Good.

In addition, in a frame in which the speech rate is smaller than the predetermined lower limit of the predetermined range, the beating feeling interval is estimated as a predetermined interval, and in a frame in which the speech rate is larger than the predetermined upper limit of the predetermined range, it is estimated from the age of the target person. With reference to the maximum heart rate, for example, the beat feeling interval is estimated using the following formula, and in a frame in which the speech rate is included in a predetermined range, the arithmetic average of the estimated values of the two beat feeling intervals is used. May be estimated as the beating feeling interval of the frame.
(220-age) x 0.9

  Further, a plurality of threshold values may be set, a plurality of predetermined ranges each having a lower limit and an upper limit may be prepared, and the beating feeling interval may be estimated for each predetermined range. In this case, it may be estimated that the larger the value included in the predetermined range, the longer the beating feeling interval.

(Estimation method 3)
Alternatively, when the speech rate in the estimation target frame is compared with the speech rate in the (previous) frame temporally prior to the estimation target frame, and the speech rate in the estimation target frame is larger, The estimated value of the beating feeling interval in the estimation target frame is made shorter (smaller) than the estimated value of the beating feeling interval in the previous frame. When the speech rate in the estimation target frame is lower, the estimated value of the beating feeling interval in the estimation target frame is made longer (larger) than the estimated value of the beating feeling interval in the previous frame.

  As described above, the beating feeling interval estimation unit 110 estimates the beating feeling interval so that the estimated value of the beating feeling interval in a frame with a high speech rate is equal to or less than the estimated value of the beating feeling interval in a frame with a low speech rate. . At this time, for at least some of the sections, there is always a relationship that the estimated value of the beating feeling interval in a frame with a high speaking speed is smaller than the estimated value of the beating feeling interval in a frame with a low speaking speed (the beating feeling interval is short). It shall hold.

(Estimation method 4)
Let F (SR) be a monotonic non-increasing function for the speech rate SR. F (SR) is a function that returns a smaller value as the speech rate SR increases. Using such a function, F (SR) may be used as an estimated value of the beating feeling interval.

  The speech rate for each frame is a value with fluctuations. In the estimation method 4, the value of F (SR) changes according to the size of the speech rate SR. Therefore, the estimated value of the beating sensation interval for each frame obtained by the estimation method 4 also has fluctuation when viewed in time series. Thereby, since the heartbeat information synthesized by the heartbeat information synthesizing unit 140 is synthesized as if the heartbeat was repeated at intervals including fluctuations, more realistic heartbeat information can be synthesized. .

  By using such estimation methods 1 to 4, without using a biosensor or the like, from information acquired by a method that does not interfere with the subject (speech signal including the speech of the subject collected by the microphone), The beating feeling interval can be estimated.

  The beating feeling interval estimation unit 110 outputs the estimated beating feeling interval for each frame to the beating feeling information synthesis unit 140.

<Beat Feeling Intensity Estimator 120>
The pulsation feeling intensity estimation unit 120 estimates the pulsation feeling intensity using the utterance speed SR output from the utterance speed calculation unit 105 (s120), and outputs the estimated value to the pulsation feeling information synthesis unit 140.

  At this time, the estimated value of the beating feeling intensity corresponding to the frame having the utterance speed SR having the first value is more than the estimated value of the beating feeling intensity corresponding to the frame having the utterance speed SR having the second value smaller than the first value. Is estimated to be large.

  That is, the pulsation feeling intensity is estimated so that the estimated value of the pulsation feeling intensity is larger in a frame with a high (fast) utterance speed SR than in a frame with a low (slow) utterance speed.

  For silent frames (that is, frames with SR = 0), it is assumed that the previous heartbeat intensity continues, and the same value as the estimated value of the heartbeat intensity of the previous non-silent frame is used. It will be applied as an estimated value.

  The following are four methods for estimating the beating feeling interval using the speech rate SR.

(Estimation method 1)
For example, in a frame in which the speech rate is equal to or lower than a predetermined threshold, the heartbeat feeling intensity is estimated as a predetermined intensity (for example, 100). In addition, in a frame in which the speech rate is greater than a predetermined threshold, the beating feeling strength is estimated to be a strength (for example, 150) greater than a predetermined strength. As the threshold value, for example, an average of the speaking rate of the subject measured in advance in a calm state may be used.

(Estimation method 2)
Alternatively, in a frame in which the speech rate is included in a predetermined range, the heartbeat feeling strength of the frame is estimated as a predetermined strength (for example, strength 100), and the speech rate is smaller than the lower limit of the predetermined range. Then, it is estimated that the heartbeat feeling intensity is weaker than a predetermined intensity (for example, intensity 50), and the heartbeat feeling intensity is stronger than the predetermined intensity in a frame in which the speech rate is larger than the upper limit of the predetermined range (for example, The intensity 150) may be estimated. The predetermined range is, for example, if the minimum value (or minimum value -α) and the maximum value (or maximum value + α) of the speech rate for each frame of the subject measured in a calm state in advance are the lower limit and the upper limit, respectively. Good.

  Further, a plurality of threshold values may be set, and a plurality of predetermined ranges each having a lower limit and an upper limit may be prepared, and the heartbeat feeling intensity may be estimated for each predetermined range. In this case, it may be estimated that the greater the value included in the predetermined range, the greater the heartbeat feeling intensity.

(Estimation method 3)
Alternatively, when the speech rate in the estimation target frame is compared with the speech rate in the (previous) frame temporally prior to the estimation target frame, and the speech rate in the estimation target frame is larger, The estimated value of the beating feeling intensity in the estimation target frame is made higher (stronger) than the estimated value of the beating feeling intensity in the previous frame. When the speech rate in the estimation target frame is lower, the estimated value of the beating feeling intensity in the estimation target frame is set lower (weaker) than the estimated value of the beating feeling intensity in the previous frame.

  As described above, the pulsation feeling intensity estimation unit 120 estimates the pulsation feeling intensity so that the estimated value of the pulsation feeling intensity in a frame with a high utterance speed is equal to or greater than the estimated value of the pulsation feeling intensity in a frame with a low utterance speed. . At this time, for at least some of the sections, there is always a relationship that the estimated value of the beating feeling intensity in the frame with a high speaking speed is larger than the estimated value of the beating feeling intensity in the frame with a low speaking speed (strong beating feeling intensity). It shall hold.

(Estimation method 4)
Let G (SR) be a monotonic non-decreasing function for the speech rate SR. G (SR) is a function that returns a larger value as the speech rate SR increases. Using such a function, G (SR) may be used as an estimated value of the heartbeat feeling intensity.

  The speech rate for each frame is a value with fluctuations. In the estimation method 4, the value of G (SR) changes according to the size of the speech rate SR. Therefore, the estimated value of the beating sensation intensity for each frame obtained by the estimation method 4 also has fluctuations when viewed in time series. Thereby, since the heartbeat information synthesized by the heartbeat information synthesizing unit 140 is synthesized with information as if the heartbeat is repeated with an intensity including fluctuation, more realistic heartbeat information can be synthesized. .

  By using such estimation methods 1 to 4, information acquired by a method that does not interfere with the subject without using a dedicated measurement device or psychological experiment, or even a biosensor, From the received speech of the subject), the heartbeat feeling intensity can be estimated.

  The beating feeling intensity estimating unit 120 outputs the estimated beating feeling intensity for each frame to the beating feeling information combining unit 140.

<Beat Feeling Information Synthesizer 140>
The beating feeling information synthesis unit 140 receives the estimated value of the beating feeling interval output from the beating feeling interval estimation unit 110 and the estimated value of the beating feeling intensity output from the beating feeling intensity estimation unit 120, and from these values. The beating feeling information is synthesized (s140) and output to the beating feeling information presenting unit 150.

  At this time, the pulsation feeling information is information for presenting pulsations repeated at the estimated value of the pulsation feeling interval, and information for presenting the pulsation intensity corresponding to the estimated value of the pulsation feeling intensity; Synthesize so that That is, the information for one heartbeat as a reference prepared in advance is repeated with the estimated value of the beating feeling interval output from the beating feeling interval estimation unit 110, and the intensity of the pulsation in each frame is determined as the beating feeling. Beat feeling information is synthesized from the estimated value of the beating feeling interval and the estimated value of the beating feeling intensity so as to be perceived by the human as an intensity corresponding to the estimated value of the beating feeling intensity of the frame estimated by the intensity estimating unit 120. .

  As the heartbeat information, visual information (video), auditory information (sound), tactile information (vibration), or the like can be used.

  (1) When visual information (video) is presented as beating feeling information in the beating feeling information presenting section 150 described later, the beating feeling information synthesizing section 140 generates a reference waveform for one beat as the beating feeling interval. A waveform repeated at the heartbeat interval estimated by the estimation unit 110 is generated. At this time, the amplitude of the heartbeat waveform included in each frame is positively correlated with the heartbeat intensity of the frame estimated by the heartbeat intensity estimation unit 120 (that is, the estimated value of the heartbeat intensity is strong). The amplitude of the waveform is converted so that the amplitude becomes larger. As a waveform for one heartbeat serving as a reference, for example, a waveform obtained by expressing an electrocardiogram waveform for one heartbeat of a human heart acquired in advance as an image is used. FIG. 3 shows an example of video when presenting visual information (video) as heartbeat information.

  (2) In the case of presenting auditory information (sound) as beating feeling information in the beating feeling information presenting section 150 described later, the beating feeling information synthesizing section 140 generates a reference sound for one heart beat as the beating feeling interval. It repeats at the beating feeling interval estimated by the estimating unit 110, and the volume has a positive correlation with the beating feeling intensity of the frame estimated by the beating feeling intensity estimating unit 120 (that is, the stronger the beating feeling intensity is, A sequence of sound signals is generated so that the volume increases. As the reference sound for one heartbeat, for example, previously acquired voice information for one human heartbeat is used.

  (3) When tactile information (vibration) is presented as beating feeling information in the beating feeling information presenting section 150 described later, the beating feeling information synthesizing section 140 generates vibrations corresponding to one heartbeat as a beating feeling interval. It is repeated at the beating feeling interval estimated by the estimation unit 110, and the vibration intensity has a positive correlation with the beating feeling intensity of the frame estimated by the beating feeling intensity estimation unit 120 (that is, the beating feeling intensity is Vibration information is generated so that the stronger the vibration, the stronger the vibration. As the vibration for one heartbeat serving as a reference, for example, a waveform obtained by expressing a waveform of one heartbeat of a human heart acquired in advance as vibration is used.

<Beat Feeling Information Presentation Unit 150>
The beating feeling information presenting unit 150 presents the beating feeling information synthesized by the beating feeling information synthesizing unit 140 in a form that can be perceived by humans such as video, vibration, and sound. For example, if it is an image, the beating feeling information presenting unit 150 is composed of a device that emits an image such as a display. If it is vibration, the beating feeling information presenting unit 150 is composed of a vibrating device such as a vibrator. The presenting unit 150 includes a device that emits sound, such as a speaker, and heartbeat information is generated as a signal for driving the devices.

<Effect>
With the above configuration, heartbeat information can be generated only from the speech rate of the target person.

<Modification>
The beating feeling information generating apparatus 100 may not include the beating feeling information presenting unit 150 and may be configured to output the beating feeling information to another device such as a display, a vibrator, or a speaker.

  The heartbeat information generating apparatus 100 may not include the microphone 101 and may be configured to receive an audio signal including the speech of the target person as an input from a microphone of another apparatus. Similarly, when a phoneme sequence and speech rate can be received as input, the beating sensation information generation device 100 may not include the speech recognition device 103 and the speech rate calculation unit 105.

  Moreover, although the beating feeling information generation apparatus 100 of the first embodiment includes the beating feeling interval estimation unit 110 and the beating feeling intensity estimation unit 120, at least one of them may be included. A configuration including at least one of the beating feeling interval estimation unit 110 and the beating feeling intensity estimation unit 120 is referred to as a beating feeling estimation unit, and the beating feeling estimation unit estimates at least one of the beating feeling interval and the beating feeling intensity. . When the beating sensation estimation unit estimates the beating feeling interval or the beating feeling intensity, the beating feeling information synthesis unit 140 is output from the estimated value of the beating feeling interval or the beating feeling intensity estimation unit 120 output from the beating feeling interval estimation unit 110. The estimated value of the beating feeling intensity is received, the beating feeling information is synthesized from the received value (s140), and is output to the beating feeling information presenting unit 150.

  At this time, the pulsation feeling information is combined so as to be information for presenting the pulsation repeated at the estimated value of the pulsation feeling interval or the pulsation intensity corresponding to the estimated value of the pulsation feeling intensity. That is, the information for one heartbeat as a reference prepared in advance is repeated with the estimated value of the beating feeling interval output from the beating feeling interval estimation unit 110, or the intensity of the beating in each frame is beat. The pulsation feeling information is synthesized from the estimated value of the pulsation feeling interval or the estimated value of the pulsation feeling intensity so as to be perceived by the human as an intensity corresponding to the estimated value of the pulsation feeling intensity of the frame estimated by the feeling intensity estimation unit 120. . For example, when only the estimated value of the beating feeling interval is used, the intensity of the pulsation may be a constant value. When only the estimated value of the pulsation feeling intensity is used, the pulsation feeling interval may be set to a constant value. Note that as the heartbeat information, visual information (video), auditory information (sound), tactile information (vibration), and the like can be used as in the first embodiment.

The heartbeat information is not limited to that described above. The beating feeling information may be a signal for blinking a single light. For example, the beating feeling interval may be represented by blinking light and the beating feeling intensity may be represented by lighting intensity. In addition, any information may be used as long as it enables humans to perceive the heartbeat interval and the heartbeat intensity.
As the beating sensation information generating apparatus 100, a personal computer, a smart phone, or the like can be considered.

<Points of second embodiment>
In the first embodiment, the correlation between the speech rate, the heartbeat interval, and the heartbeat intensity is used. In the second embodiment, by using the correlation between the fundamental frequency of speech, the heartbeat interval and the heartbeat intensity, the approximate heartbeat interval and heartbeat intensity are estimated from the basic frequency, and the estimated value of the heartbeat interval is estimated. Heartbeat information is synthesized and presented from the estimated value of heartbeat intensity.

<Second Embodiment: Utilizing Fundamental Frequency>
A description will be given centering on differences from the first embodiment.

  FIG. 4 shows a functional block diagram of the beating feeling information generating apparatus 200 according to the second embodiment. FIG. 5 shows an example of the processing flow.

  In the second embodiment, instead of the speech speed, the heartbeat interval and the heartbeat intensity are estimated from the fundamental frequency of the speech. Generally, the fundamental frequency when a person is nervous or excited tends to be higher than the fundamental frequency during normal times. When the person is nervous or excited, the heartbeat interval (beat interval) tends to be short and the heartbeat intensity tends to increase. Therefore, the second embodiment estimates the beating feeling interval and beating feeling intensity based on the fundamental frequency.

  The beating feeling information generation device 200 of the second embodiment includes a fundamental frequency analysis unit 205 instead of the speech recognition device 103 and the speech rate calculation unit 105 of the beating feeling information generation device 100 of the first embodiment. Also, the processing of the beating feeling interval estimation unit 210 and the beating feeling intensity estimation unit 220 are different. Only processing different from that of the first embodiment will be described, and description of overlapping parts will be omitted.

<Basic frequency analysis unit 205>
The fundamental frequency analysis unit 205 obtains and outputs a fundamental frequency from the input audio signal in frame units. As a calculation method of the fundamental frequency, for example, a known technique such as the modified correlation method of Reference 3 may be used.
(Reference 3) Furui, “Digital Audio Processing”, Tokai University Press, 1985, pp.69-72.
The fundamental frequency obtained by the fundamental frequency analysis unit 205 is input to the beating feeling interval estimation unit 210 and the beating feeling intensity estimation unit 220.

<Beat Feeling Interval Estimation Unit 210>
The beating feeling interval estimation unit 210 estimates the beating feeling interval (that is, the heartbeat interval) using the fundamental frequency (S210), and outputs the estimated value to the beating feeling information synthesis unit 140.

  At this time, the estimated value of the beating feeling interval corresponding to the frame whose fundamental frequency is the first value is smaller than the estimated value of the beating feeling interval corresponding to the frame whose basic frequency is the second value smaller than the first value. The beating feeling interval is estimated so as to be (short).

  That is, the heartbeat feeling interval is estimated so that the estimated value of the heartbeat feeling interval is smaller (shorter) in a frame having a higher (higher) fundamental frequency than in a frame having a lower (lower) fundamental frequency. For silent frames, it is assumed that the immediately preceding beating feeling interval continues, and the same value as the estimated value of the beating feeling interval of the immediately preceding non-silent frame is applied as the estimated value of the beating feeling interval. .

  The following are four methods for estimating the beating feeling interval using the fundamental frequency.

(Estimation method 1)
For example, in a frame in which the fundamental frequency is equal to or less than a predetermined threshold, the beating feeling interval is estimated as a predetermined interval (eg, 1200 ms). For example, the predetermined interval may be calculated from the heart rate of the subject measured in a calm state in advance. Further, in a frame in which the fundamental frequency is larger than a predetermined threshold, the beating feeling interval is estimated to be shorter than the predetermined interval (for example, 750 ms). As the threshold value, for example, an average of the fundamental frequency of the subject measured in advance in a calm state may be used.

(Estimation method 2)
Alternatively, in a frame in which the fundamental frequency is included within a predetermined range, the beating feeling interval of the frame is estimated as a predetermined interval (for example, 1200 ms), and in a frame in which the fundamental frequency is less than the lower limit of the predetermined range. It is estimated that the heartbeat interval is longer than a predetermined interval (for example, 1400 ms), and the heartbeat interval is shorter than the predetermined interval (for example, 750 ms) in a frame whose fundamental frequency is larger than the upper limit of the predetermined range. May be estimated. The predetermined range is, for example, if the minimum value (or minimum value -α) and the maximum value (or maximum value + α) of the fundamental frequency for each frame of the subject measured in a calm state in advance are set as the lower limit and the upper limit, respectively. Good.

Also, in a frame where the fundamental frequency is smaller than the lower limit of the predetermined range, the beating feeling interval is estimated as a predetermined interval, and in a frame where the fundamental frequency is larger than the upper limit of the predetermined range, it is estimated from the age of the subject. With reference to the maximum heart rate, for example, the heartbeat interval is estimated using the following formula, and the arithmetic average of the estimated values of the two heartbeat intervals in a frame in which the fundamental frequency falls within a predetermined range. May be estimated as the beating feeling interval of the frame.
(220-age) x 0.9

  Further, a plurality of threshold values may be set, a plurality of predetermined ranges each having a lower limit and an upper limit may be prepared, and the beating feeling interval may be estimated for each predetermined range. In this case, it may be estimated that the larger the value included in the predetermined range, the longer the beating feeling interval.

(Estimation method 3)
Alternatively, when the fundamental frequency in the estimation target frame is compared with the fundamental frequency in the temporally previous (previous) frame before the estimation target frame, and the fundamental frequency in the estimation target frame is larger, The estimated value of the beating feeling interval in the estimation target frame is made shorter (smaller) than the estimated value of the beating feeling interval in the previous frame. When the fundamental frequency in the estimation target frame is smaller, the estimated value of the beating feeling interval in the estimation target frame is made longer (larger) than the estimated value of the beating feeling interval in the previous frame.

  In this way, the heartbeat interval estimation unit 210 estimates the heartbeat interval so that the estimated value of the heartbeat interval in a frame with a high fundamental frequency is equal to or less than the estimated value of the heartbeat interval in a frame with a low fundamental frequency. . At this time, for at least some of the sections, there is always a relationship that the estimated value of the beating feeling interval in a frame with a high fundamental frequency is smaller than the estimated value of the beating feeling interval in a frame with a low basic frequency (the beating feeling interval is short). It shall hold.

(Estimation method 4)
Let H (F0) be a monotonic non-increasing function related to the fundamental frequency F0. H (F0) is a function that returns a smaller value as the fundamental frequency F0 is larger. Using such a function, H (F0) may be used as an estimated value of the heartbeat interval.

  The fundamental frequency for each frame is a value with fluctuation. In the estimation method 4, since the value of H (F0) changes according to the magnitude of the fundamental frequency F0, the estimated value of the beating sensation interval for each frame obtained by the estimation method 4 also has fluctuation when viewed in time series. Thereby, since the heartbeat information synthesized by the heartbeat information synthesizing unit 140 is synthesized as if the heartbeat was repeated at intervals including fluctuations, more realistic heartbeat information can be synthesized. .

  By using such estimation methods 1 to 4, without using a biosensor or the like, from information acquired by a method that does not interfere with the subject (speech signal including the speech of the subject collected by the microphone), The beating feeling interval can be estimated.

  The beating feeling interval estimation unit 210 outputs the estimated beating feeling interval for each frame to the beating feeling information synthesis unit 140.

<Beat Feeling Intensity Estimator 220>
The pulsation feeling intensity estimation unit 220 estimates the pulsation feeling intensity using the fundamental frequency obtained by the fundamental frequency analysis unit 205 (s220), and outputs the estimated value to the pulsation feeling information synthesis unit 140.

  At this time, the estimated value of the beating feeling intensity corresponding to the frame whose fundamental frequency is the first value is larger than the estimated value of the beating feeling intensity corresponding to the frame whose basic frequency is the second value smaller than the first value. In this way, the pulsation intensity is estimated. That is, the pulsation feeling intensity is estimated so that the estimated value of the pulsation feeling intensity of the frame having a higher fundamental frequency is larger (stronger) than the estimated value of the pulsation feeling intensity of the frame having a lower fundamental frequency.

  For silent frames, it is assumed that the immediately preceding heartbeat feeling intensity continues, and the same value as the estimated value of the heartbeat feeling intensity of the immediately preceding non-silent frame is applied as the estimated value of the heartbeat feeling intensity. .

  The beating sensation intensity estimation unit 220 estimates the beating sensation intensity for each frame based on the fundamental frequency for each frame.

  The following are four methods for estimating the pulsation intensity using the fundamental frequency.

(Estimation method 1)
For example, in a frame in which the fundamental frequency is equal to or less than a predetermined threshold, the heartbeat feeling intensity is estimated as a predetermined intensity (for example, 100). In addition, in a frame in which the fundamental frequency is larger (higher) than a predetermined threshold, the heartbeat feeling intensity is estimated to be an intensity (for example, 150) larger than a predetermined intensity.

(Estimation method 2)
Alternatively, in a frame in which the fundamental frequency is included within a predetermined range, the heartbeat feeling intensity of the frame is estimated as a predetermined intensity (for example, intensity 100), and the fundamental frequency is greater (higher) than the predetermined range. In the frame, it is estimated that the heartbeat feeling intensity is stronger than the predetermined intensity (for example, intensity 150), and in the frame whose fundamental frequency is smaller (lower) than the predetermined range, the heartbeat feeling intensity is weaker than the predetermined intensity. (For example, intensity 50) may be estimated.

  Further, a plurality of threshold values may be set, and a plurality of predetermined ranges each having a lower limit and an upper limit may be prepared, and the heartbeat feeling intensity may be estimated for each predetermined range. In this case, it may be estimated that the greater the value included in the predetermined range, the greater the heartbeat feeling intensity.

  The thresholds of the estimation method 1 and the estimation method 2 and the upper and lower limits of the predetermined range may be determined by using the same method as in the first embodiment (however, the average of the fundamental frequency is used instead of the average of the speech rate). .

(Estimation method 3)
Alternatively, when the fundamental frequency in the estimation target frame is compared with the fundamental frequency in the temporally previous (previous) frame before the estimation target frame, and the fundamental frequency in the estimation target frame is larger, The estimated value of the beating feeling intensity in the estimation target frame is made larger (stronger) than the estimated value of the beating feeling intensity in the previous frame. In addition, when the fundamental frequency in the estimation target frame is smaller, the estimated value of the beating feeling intensity in the estimation target frame is made lower (weaker) than the estimated value of the beating feeling intensity in the previous frame.

  As described above, the pulsation feeling intensity estimation unit 220 estimates the pulsation feeling intensity so that the estimated value of the pulsation feeling intensity in a frame with a high fundamental frequency is equal to or greater than the estimated value of the pulsation feeling intensity in a frame with a low fundamental frequency. . At this time, for at least some of the sections, there is always a relationship that the estimated value of the heartbeat feeling intensity in the frame having a high fundamental frequency is larger than the estimated value of the heartbeat feeling intensity in the frame having a low fundamental frequency (the heartbeat feeling intensity is strong). It shall hold. In other words, the beating sensation intensity estimation unit 220 estimates the beating sensation intensity in a frame in which the estimated value of the beating sensation intensity in a frame whose fundamental frequency is the first value is a second value whose fundamental frequency is smaller than the first value. The beating feeling intensity is estimated so as to be larger than the value.

(Estimation method 4)
Let J (F0) be a monotonic non-decreasing function for the fundamental frequency F0. J (F0) is a function that returns a larger value as the fundamental frequency F0 is larger. Using such a function, J (F0) may be used as an estimated value of the pulsation feeling intensity.

  The fundamental frequency for each frame is a value with fluctuation. In the estimation method 4, the value of J (F0) changes in accordance with the magnitude of the fundamental frequency F0. Therefore, the estimated value of the heartbeat sensation intensity for each frame obtained by the estimation method 4 also has fluctuations when viewed in time series. Thereby, since the heartbeat information synthesized by the heartbeat information synthesizing unit 140 is synthesized with information as if the heartbeat is repeated with an intensity including fluctuation, more realistic heartbeat information can be synthesized. .

  By using such estimation methods 1 to 4, information acquired by a method that does not interfere with the subject without using a dedicated measurement device or psychological experiment, or even a biosensor, From the received speech of the subject), the heartbeat feeling intensity can be estimated.

  The beating feeling intensity estimation unit 220 outputs the estimated beating feeling intensity for each frame to the beating feeling information synthesis unit 140.

<Effect>
By setting it as such a structure, the effect similar to 1st embodiment can be acquired. In addition, you may combine this embodiment and the modification of 1st embodiment. For example, the beating feeling information generation apparatus 200 of the present embodiment includes the beating feeling interval estimation unit 210 and the beating feeling intensity estimation unit 220, but at least one of them may be included.

<Third embodiment>
A description will be given centering on differences from the first embodiment.
FIG. 6 shows a functional block diagram of the beating feeling information generating apparatus 300 according to the third embodiment. FIG. 7 shows an example of the processing flow.

  In this embodiment, the beat feeling interval and the beat feeling intensity are estimated by combining the first embodiment and the second embodiment. That is, the beating feeling interval estimation unit 310 and the beating feeling intensity estimation unit 320 of the beating feeling information generation apparatus 300 according to the present embodiment estimate the beating feeling interval and the beating feeling intensity based on both the speech rate and the fundamental frequency, respectively. To do.

  The difference from the first embodiment is the processing of the “fundamental frequency analysis unit 205”, “beating interval estimation unit 310”, and “beating feeling intensity estimation unit 320”. Among these, since the fundamental frequency analysis unit 205 is the same as that of the second embodiment, only the processing of the beating feeling interval estimation unit 310 and the beating feeling intensity estimation unit 320 will be described below, and description of overlapping portions will be omitted.

<Beat Feeling Interval Estimation Unit 310>
The beating sensation interval estimation unit 310 estimates the beating sensation interval (beat interval) using the fundamental frequency obtained by the fundamental frequency analysis unit 205 and the utterance speed obtained by the utterance speed calculation unit 105 (S310). The value is output to the beating feeling information synthesis unit 140.
At this time, the beat feeling intensity is estimated so that the estimated value of the beat feeling interval becomes smaller (shorter) as the fundamental frequency is higher and the speech speed is faster.

  The following are three methods for estimating the beating feeling interval using the fundamental frequency and the speech rate.

(Estimation method 1)
For example, in a frame in which the speech rate is equal to or lower than a predetermined first threshold and the fundamental frequency is equal to or lower than a predetermined second threshold, the beating feeling interval is estimated as a predetermined interval (for example, 1200 ms). In addition, in a frame in which the fundamental frequency is larger than the predetermined second threshold, it is estimated that the heartbeat interval is short (for example, 750 ms) (regardless of the speech rate). Further, in a frame in which the fundamental frequency is equal to or lower than a predetermined second threshold and the speech rate is higher than the predetermined first threshold, it is estimated that the heartbeat interval is slightly short (for example, 1000 ms). A summary of this relationship is shown in FIG. 8a. The longest estimated value is calculated from the heart rate of the subject measured in advance in a calm state, and the shortest estimated value is estimated from the age of the subject. With reference to the maximum heart rate, for example, it is calculated using the following mathematical formula, and the intermediate estimated value is an arithmetic average of the estimated values of the two heartbeat intervals.
(220-age) x 0.9
In this example, in a frame in which the fundamental frequency is greater than the predetermined second threshold, the heartbeat interval is short (regardless of the speech speed), but the speech speed is greater than the first threshold and the fundamental The frame is set so that the estimated value of the beating sensation interval is shorter in a frame whose frequency is higher than the second threshold than in a frame whose speech speed is equal to or lower than the first threshold and whose fundamental frequency is higher than the second threshold. (For example, the beating feeling intervals are estimated to be 700 ms and 800 ms, respectively).

  That is, the heartbeat interval of a frame in which the speech rate is greater than the first threshold and the fundamental frequency is greater than the second threshold, the frame in which the speech rate is less than or equal to the first threshold and the fundamental frequency is less than or equal to the second threshold. The beating feeling interval is estimated to be shorter than the beating feeling interval.

(Estimation method 2)
Alternatively, the speech rate and the fundamental frequency in the estimation target frame are compared with the speech rate and the fundamental frequency in a frame that is temporally earlier (past) than the estimation target frame, and the speech rate or the fundamental frequency in the estimation target frame is compared. When at least one of the fundamental frequencies is large, the estimated value of the beating feeling interval in the estimation target frame is made shorter (smaller) than the estimated value of the beating feeling interval in the previous frame. In addition, when at least one of the speech rate and the fundamental frequency in the estimation target frame is small, the estimation value of the beating feeling interval in the estimation target frame is longer (larger) than the estimation value of the beating feeling interval in the previous frame. ) At this time, when both the speech rate and the fundamental frequency of the estimation target frame are larger than the speech rate and the fundamental frequency of the previous frame, either the speech rate or the fundamental frequency of the estimation target frame is the previous one. A shorter estimated value of the heartbeat interval may be set than when either the frame speech rate or the fundamental frequency is larger. Similarly, if both the speech rate and the fundamental frequency of the estimation target frame are lower than the speech rate and the fundamental frequency of the previous frame, either the speech rate or the fundamental frequency of the estimation target frame is the previous one. A longer estimated value of the beating sensation interval may be set than in the case where it is smaller than either the speech rate of the frame or the fundamental frequency.

  As described above, the beat feeling interval estimation unit 310 has an estimated value of the beating feeling interval in a frame with a large speech rate and fundamental frequency smaller (shorter) than an estimated value of the beating feeling interval in a frame with a small speech rate and fundamental frequency. The beating feeling interval is estimated as follows. In other words, the beating sensation interval estimation unit 310 has an estimated value of the beating sensation interval in a frame in which the speech speed is the first value and the fundamental frequency is the third value, as a second value whose utterance speed is smaller than the first value. The beating feeling interval is estimated so as to be smaller (shorter) than the estimated value of the beating feeling interval in the frame whose fundamental frequency is the fourth value smaller than the third value. Alternatively, for at least a part of the interval, the estimated value of the beating feeling interval in the frame in which the speech speed is the first value is more than the estimated value of the beating feeling interval in the frame in which the speech speed is the second value smaller than the first value. And the estimated value of the beating sensation interval in a frame whose fundamental frequency is the third value is a fourth value whose fundamental frequency is smaller than the third value in at least some sections. The beating feeling interval is estimated so as to include a case that is smaller (shorter) than the estimated value of the beating feeling interval in a certain frame.

(Estimation method 3)
The pulsation feeling interval may be estimated using the speech rate and the fundamental frequency, and the two pulsation feeling intervals may be weighted and added to estimate the final pulsation feeling interval.
For example, the first beating sensation interval BD1 is estimated using the speech rate by the estimation method of the first embodiment. The second heartbeat interval BD2 is estimated using the fundamental frequency by the estimation method of the second embodiment.

The first beating feeling interval BD1 and the second beating feeling interval BD2 are weighted and added as in the following equation to obtain a third beating feeling interval BD3, which is output as an output value of the beating feeling interval estimation unit 310.
However, from Reference Document 1, it is considered that the fundamental frequency has a greater influence on the heartbeat interval than the speech speed, so α is set to a value larger than 0.5 and 0.5 <α <1.

  The beating feeling interval estimation unit 310 outputs the estimated beating feeling interval for each frame to the beating feeling information synthesis unit 140.

<Beat Feeling Intensity Estimator 320>
The pulsation feeling intensity estimation unit 320 estimates the pulsation feeling intensity using the fundamental frequency obtained by the fundamental frequency analysis unit 205 and the utterance speed obtained by the utterance speed calculation unit 105 (s320), and uses the estimated value as the pulsation feeling information. The data is output to the synthesis unit 140.

  At this time, the heartbeat feeling intensity is estimated so that the estimated value of the heartbeat feeling intensity increases as the fundamental frequency increases and the speech rate increases.

The following are three methods for estimating the pulsation intensity using the fundamental frequency and the speech rate.
(Estimation method 1)
For example, in a frame in which the speech rate is equal to or lower than a predetermined first threshold and the fundamental frequency is equal to or lower than a predetermined second threshold, the heartbeat feeling intensity is estimated as a predetermined intensity (for example, 100). In addition, in a frame in which the fundamental frequency is larger than a predetermined second threshold, it is estimated that the heartbeat feeling strength is strong (for example, strength 150) (regardless of the speech rate). Further, in a frame in which the fundamental frequency is equal to or lower than a predetermined second threshold and the speech rate is higher than the predetermined first threshold, it is estimated that the heartbeat feeling intensity is slightly strong (for example, intensity 120). A summary of this relationship is shown in FIG. 8b. In this example, in a frame in which the fundamental frequency is greater than the second threshold, the heartbeat feeling strength is strong (regardless of the speech speed), but the speech speed is greater than the first threshold and the fundamental frequency is the first. A frame larger than 2 thresholds may be set so that the pulsation intensity is stronger than a frame whose utterance speed is equal to or lower than a predetermined first threshold and whose fundamental frequency is higher than a predetermined second threshold ( For example, the pulsation intensity is estimated to be 160 and 140, respectively).

  In other words, the heartbeat feeling intensity of a frame in which the speech rate is greater than the first threshold and the fundamental frequency is greater than the second threshold, the frame in which the speech rate is less than or equal to the first threshold and the fundamental frequency is less than or equal to the second threshold. The pulsation feeling intensity is estimated to be larger than the pulsation feeling intensity.

(Estimation method 2)
Alternatively, the speech rate and the fundamental frequency in the estimation target frame are compared with the speech rate and the fundamental frequency in a frame that is temporally earlier (past) than the estimation target frame, and the speech rate or the fundamental frequency in the estimation target frame is compared. When at least one of the fundamental frequencies is large, the estimated value of the beating feeling intensity in the estimation target frame is made higher (stronger) than the estimated value of the beating feeling intensity in the previous frame. In addition, when at least one of the speech rate and the fundamental frequency in the estimation target frame is small, the estimation value of the beating feeling intensity in the estimation target frame is lower (weaker) than the estimation value of the beating feeling intensity in the previous frame. ) At this time, when both the speech rate and the fundamental frequency of the estimation target frame are larger than the speech rate and the fundamental frequency of the previous frame, either the speech rate or the fundamental frequency of the estimation target frame is the previous one. A larger estimated value of the pulsation feeling intensity may be set than in the case where it is higher than either the speech rate of the frame or the fundamental frequency. Similarly, if both the speech rate and the fundamental frequency of the estimation target frame are lower than the speech rate and the fundamental frequency of the previous frame, either the speech rate or the fundamental frequency of the estimation target frame is the previous one. A smaller estimated value of the pulsation sensation intensity may be set than when the utterance speed or the fundamental frequency of the frame is smaller than either one.

  As described above, the beating sensation intensity estimation unit 320 is configured so that the estimated value of the beating feeling intensity in a frame with a large speech rate and fundamental frequency is larger than the estimated value of the beating feeling intensity in a frame with a small speech rate and fundamental frequency. Estimate the pulsation intensity. In other words, the beating sensation intensity estimation unit 320 has an estimated value of beating sensation intensity in a frame in which the utterance speed is the first value and the fundamental frequency is the third value, as a second value whose utterance speed is smaller than the first value. The beating sensation intensity is estimated so as to be larger than the estimated value of the beating sensation intensity in the frame whose fundamental frequency is the fourth value smaller than the third value. Alternatively, for at least some of the sections, the estimated value of the beating feeling intensity in the frame in which the speech speed is the first value is more than the estimated value of the beating feeling intensity in the frame in which the speech speed is the second value smaller than the first value. In the frame in which the estimated value of the beating sensation intensity in the frame in which the fundamental frequency is the third value is the fourth value in which the fundamental frequency is smaller than the third value. The beating feeling intensity is estimated so as to include a case where the estimated value is larger than the estimated value of the beating feeling intensity.

(Estimation method 3)
The pulsation feeling intensity may be estimated using the speech rate and the fundamental frequency, and the final pulsation feeling intensity may be estimated by weighting and adding the two pulsation feeling intensities.

  For example, the first beating sensation intensity BS1 is estimated using the speech rate by the estimation method of the first embodiment. The second heartbeat feeling intensity BS2 is estimated using the fundamental frequency by the estimation method of the second embodiment.

The first pulsation feeling intensity BS1 and the second pulsation feeling intensity BS2 are weighted and added as in the following equation to obtain a third pulsation feeling intensity BS3, which is output as an output value of the pulsation feeling intensity estimation unit 310.
However, from Reference Document 1, it is considered that the fundamental frequency has a greater influence on the pulsation intensity than the speech speed, so β is set to a value larger than 0.5 and 0.5 <β <1.

  The beating feeling intensity estimation unit 320 outputs the estimated beating feeling intensity for each frame to the beating feeling information synthesis unit 140.

<Effect>
By setting it as such a structure, the effect similar to 1st embodiment can be acquired. Furthermore, by considering not only the speech rate but also the fundamental frequency, more accurate heartbeat information can be obtained. In addition, you may combine this embodiment and the modification of 1st embodiment. For example, the beating feeling information generation apparatus 300 of the present embodiment includes the beating feeling interval estimation unit 310 and the beating feeling intensity estimation unit 320, but at least one of them may be included.

<Fourth embodiment>
In the fourth embodiment, a beating feeling information distribution system 50 using the beating feeling information generation device according to any one of the first to third embodiments will be described.
The beating sensation information distribution system 50 distributes the beating sensation of a certain person (hereinafter also referred to as “first person”, for example, a performer of the stage) to others (for example, the audience).

  9 shows a configuration example of the heartbeat information distribution system 50, FIG. 10 shows a flowchart thereof, and FIG. 11 shows a usage example thereof. The heartbeat information distribution system 50 includes a distribution server 52 and N heartbeat information generators 100-n. However, n = 1, 2,..., N. As the heartbeat information generating apparatus 100-n, a personal computer, a smartphone, or the like can be considered as shown in FIG.

  On the stage, a microphone 101 for collecting the voice of the first person is installed. The microphone 101 may be attached to the neck of the first person, or may be installed somewhere on the stage.

  The first person's utterance is picked up by the microphone 101 (S 101), and an audio signal including the first person's utterance is transmitted to the distribution server 52.

  The distribution server 52 receives the audio signal and distributes it to the beating sensation information generation device 100-n possessed by N others (S52).

  The heartbeat information generating device 100-n is the heartbeat information generating device according to any one of the first to third embodiments, and the voice recognition device 103 of the heartbeat information generating device 100-n and / or the fundamental frequency analysis. The audio signal transmitted from the distribution server 52 is input to the unit 205.

  The beating feeling information generating apparatus 100-n synthesizes beating feeling information using an audio signal and presents the beating feeling information in a form that can be perceived by others (S103 and S105, and / or S205 and S110 ( Or S210 or S310), S120 (or S220 or S320), S140, S150).

  In this heartbeat information distribution system 50, since the information to be transmitted / received is only an audio signal, the volume of data to be transmitted / received can be reduced, and the heartbeat information generation device 100-n possessed by another person can also be simplified. It is possible to synthesize heartbeat information only by simple calculation and to share the heartbeat feeling of the first person among N people. Further, since there is no need to attach a biometric sensor to the first person, the first person is not restrained and the first person's operation (acting, etc.) is not hindered.

  In this embodiment, S103 and S105, and / or S205, S110 (or S210 or S310), S120 (or S220 or S320), S140, and S150 are performed on the heartbeat information generating apparatus 100-n side. However, each process may be performed on the distribution server 52 side. For example, the distribution server 52 includes the speech recognition device 103, the speech rate calculation unit 105, and / or the fundamental frequency analysis unit 205, and uses the speech rate and / or fundamental frequency in place of the speech signal to the heartbeat information generating device 100-n. You may distribute. At the time of distribution, the distribution server distributes the speech rate and / or the basic frequency in L or R of the video / audio. In order to put the speech rate and / or the fundamental frequency on one of the left and right channel video and audio signals, the speech rate and / or the fundamental frequency is encoded and distributed. The heartbeat information generating apparatus 100-n decodes the encoded speech rate and / or fundamental frequency, synthesizes the heartbeat information based on the decoded information, and generates the heartbeat information in a form that can be perceived by others. Present. Note that since the speech speed and / or the fundamental frequency is on one of the left and right channels, the audio of the video is monaural. As a result, the heartbeat information can be distributed in an existing video / audio line. In addition, the distribution server 52 includes the voice recognition device 103, the speech rate calculation unit 105 and / or the fundamental frequency analysis unit 205, the heartbeat interval estimation unit 110 or 210, and the heartbeat intensity estimation unit 120 or 220. You may deliver a space | interval and a heartbeat feeling intensity | strength to the heartbeat feeling information generation apparatus 100-n. In addition, the distribution server 52 includes a speech recognition device 103, an utterance speed calculation unit 105, and / or a fundamental frequency analysis unit 205, a beating feeling interval estimation unit 110, 210, or 310, and a beating feeling intensity estimation unit 120, 220, or 320. The pulsation feeling information synthesizing unit 140 may be provided, and the pulsation feeling information may be distributed to the pulsation feeling information generation device 100-n. The heartbeat information generating apparatus 100-n side may perform necessary processing according to the received information, and may include at least a portion necessary for the processing. For example, when the heartbeat information generating device 100-n receives the speech rate and / or the fundamental frequency, the heartbeat information generating device 100-n includes the heartbeat interval estimation unit 110, 210, or 310, and the heartbeat strength estimation unit. 120, 220 or 320, at least a beating feeling information synthesizing section 140 and a beating feeling information presenting section 150 are provided, and each process of S110 (or S210 or S310), S120 (or S220 or S320), S140, and s150 is performed.

  The distribution server 52 receives an audio signal picked up by a microphone, and uses this information as it is or information generated from the audio signal ((1) speaking speed, (2) basic frequency, (3) speaking speed and basic The combination of the frequency, (4) the estimated value of the beating feeling interval and the estimated value of the beating feeling intensity, or (5) the beating feeling information) is delivered to the beating feeling information generating apparatus 100-n. The beating feeling information generating device 100-n includes information for generating beating feeling information or beating feeling information ((1) speaking speed, (2) basic frequency, (3) combination of speaking speed and basic frequency, (4) Estimated value of beating feeling interval and estimated value of beating feeling intensity) and generating beating feeling information as necessary. The beating feeling information presenting unit 150 of the beating feeling information generating device 100-n Present information in a form that can be perceived by humans, such as video, vibration, and sound. What is necessary is just to change a process part suitably according to the information amount transmitted / received and the calculation amount by the side of the delivery server 52 or the heartbeat information generation apparatus 100-n.

<Other variations>
The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

<Program and recording medium>
In addition, various processing functions in each device described in the above embodiments and modifications may be realized by a computer. In that case, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its storage unit. When executing the process, this computer reads the program stored in its own storage unit and executes the process according to the read program. As another embodiment of this program, a computer may read a program directly from a portable recording medium and execute processing according to the program. Further, each time a program is transferred from the server computer to the computer, processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program includes information provided for processing by the electronic computer and equivalent to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In addition, although each device is configured by executing a predetermined program on a computer, at least a part of these processing contents may be realized by hardware.

Claims (12)

The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
A pulsation feeling information estimation unit that estimates at least one of the pulsation feeling interval and the pulsation feeling intensity from the utterance of the subject person;
A pulsation feeling information combining unit that synthesizes pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity;
The beating feeling information estimation unit
Using the speed of the subject's utterance, the estimated value of the beating feeling interval corresponding to the section in which the speed of the subject's utterance is the first value is the second value in which the speed of the subject's utterance is smaller than the first value. A beating feeling interval estimation unit that estimates the beating feeling interval so as to be smaller than the estimated value of the beating feeling interval corresponding to the section,
Using the speed of the subject's utterance, the estimated value of the beating feeling intensity corresponding to the section in which the speed of the subject's utterance is the first value is the second value where the speed of the subject's utterance is smaller than the first value. Including at least one of the beating feeling intensity estimation unit for estimating the beating feeling intensity so as to be larger than the estimated value of the beating feeling intensity corresponding to the section,
Heartbeat information generator. The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
A pulsation feeling information estimation unit that estimates at least one of the pulsation feeling interval and the pulsation feeling intensity from the utterance of the subject person;
A pulsation feeling information combining unit that synthesizes pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity;
The beating feeling information estimation unit
With 0.5 <α <1, and using the subject's utterance speed, the estimated value of the beating sensation interval corresponding to the section in which the subject's utterance speed is the first value is the target utterance speed. The first beating feeling interval is estimated to be smaller than the estimated value of the beating feeling interval corresponding to the interval that is the second value smaller than the value, and the subject's utterance is used by using the basic frequency of the subject's utterance. The estimated value of the beating feeling interval corresponding to the section in which the estimated value of the beating feeling interval corresponding to the section in which the fundamental frequency of the subject is the first value is the second value in which the fundamental frequency of the subject's utterance is smaller than the first value. The second pulsation feeling interval is estimated to be smaller than the first pulsation feeling interval estimated value BD1 and the second pulsation feeling interval estimation value BD2.
A beating sensation interval estimation unit that estimates as
Using 0.5 <β <1 and the subject's utterance speed, the estimated value of the beating sensation intensity corresponding to the section where the subject's utterance speed is the first value is the subject's utterance speed is the first. The first pulsation feeling intensity is estimated to be larger than the estimated value of the pulsation feeling intensity corresponding to the second value interval that is smaller than the value, and the target person's utterance is used by using the basic frequency of the utterance of the target person. The estimated value of the beating feeling intensity corresponding to the section in which the fundamental frequency of the subject's utterance is the second value smaller than the first value is the estimated value of the beating feeling intensity corresponding to the section in which the basic frequency is the first value. The second pulsation feeling intensity is estimated to be larger than the first pulsation feeling intensity estimated value BS1 and the second pulsation feeling intensity estimated value BS2.
Including a beating sensation intensity estimation unit
Heartbeat information generator. The beating feeling information generating device according to claim 1 or 2 ,
The beating feeling interval estimation unit, for a silent frame, the same value as the estimated value of the beating feeling interval of the immediately preceding non-silent frame is set as the estimated value of the beating feeling interval,
For the silent frame, the beating feeling intensity estimation unit uses the same value as the estimated value of the beating feeling intensity of the immediately preceding non-silent frame as the estimated value of the beating feeling intensity.
Heartbeat information generator. The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
A pulsation feeling information estimation unit that estimates at least one of the pulsation feeling interval and the pulsation feeling intensity from the utterance of the subject person;
A pulsation feeling information combining unit that synthesizes pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity;
The beating feeling information estimation unit
Using the fundamental frequency of the subject's utterance, the estimated value of the heartbeat interval corresponding to the section where the fundamental frequency of the subject's utterance is the first value is smaller than the first value of the fundamental frequency of the subject's utterance Second value
A beating feeling interval estimation unit that estimates the beating feeling interval so as to be smaller than the estimated value of the beating feeling interval corresponding to the section,
Using the fundamental frequency of the subject's utterance, the estimated value of the heartbeat sensation intensity corresponding to the section where the fundamental frequency of the subject's utterance is the first value is smaller than the first value of the subject's utterance. Including at least one of the pulsation feeling intensity estimation unit for estimating the pulsation feeling intensity so as to be larger than the estimated value of the pulsation feeling intensity corresponding to the section which is the second value,
The beating feeling interval estimation unit, for a silent frame, the same value as the estimated value of the beating feeling interval of the immediately preceding non-silent frame is set as the estimated value of the beating feeling interval,
For the silent frame, the beating feeling intensity estimation unit uses the same value as the estimated value of the beating feeling intensity of the immediately preceding non-silent frame as the estimated value of the beating feeling intensity.
Heartbeat information generator. The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
A pulsation feeling information estimation unit that estimates at least one of the pulsation feeling interval and the pulsation feeling intensity from the utterance of the subject person;
A pulsation feeling information combining unit that synthesizes pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity;
Including a beating feeling information presenting section that presents beating feeling information synthesized from at least one of the estimated value of the beating feeling interval and the estimated value of the beating feeling intensity,
The heartbeat information presented by the heartbeat information presentation unit is :
The sound for one heart beat prepared in advance is a sound that is repeated with the estimated value of the beating feeling interval, and / or the sound volume is increased as the estimated value of the beating feeling intensity increases. which is the difference between the sound,
Heartbeat information generator. The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
A pulsation feeling information estimation unit that estimates at least one of the pulsation feeling interval and the pulsation feeling intensity from the utterance of the subject person;
A pulsation feeling information combining unit that synthesizes pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity;
Including a beating feeling information presenting section that presents beating feeling information synthesized from at least one of the estimated value of the beating feeling interval and the estimated value of the beating feeling intensity,
The heartbeat information presented by the heartbeat information presentation unit is :
A waveform of a heartbeat prepared in advance is an image of a waveform repeated with the estimated value of the beating feeling interval, and / or the larger the estimated value of the beating feeling intensity, the larger the amplitude of the waveform. It is a video of the waveform synthesized in the
Heartbeat information generator. A distribution system including a distribution server and a heartbeat information generating device,
The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
The distribution server estimates at least one of a beating feeling interval and a beating feeling intensity using an audio signal including a speech of a subject collected by a microphone, and the estimated value of the beating feeling interval and the beating feeling Distributing at least one of the estimated intensity value to the beating sensation information generating device,
The beating feeling information generating device
A pulsation feeling information presentation unit that synthesizes and presents pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity received from the distribution server;
The heartbeat information presented by the heartbeat information presentation unit is:
(1) The waveform of one heartbeat prepared in advance is a waveform that is repeated with the estimated value of the beating feeling interval, and / or the larger the estimated value of the beating feeling intensity, the larger the amplitude of the waveform. It is a waveform synthesized as
And / or
(2) The sound for one heart beat prepared in advance is a sound that is repeated with the estimated value of the beating feeling interval, and / or the sound volume increases as the estimated value of the beating feeling strength increases. Is a synthesized sound like
And / or
(3) The vibration of one heartbeat prepared in advance is a vibration that is repeated with the estimated value of the beating feeling interval, and / or the intensity of the vibration increases as the estimated value of the beating feeling intensity increases. Ri synthesized vibration der so,
The distribution server
Using the speed of the subject's utterance, the estimated value of the beating feeling interval corresponding to the section in which the speed of the subject's utterance is the first value is the second value in which the speed of the subject's utterance is smaller than the first value. Corresponds to the interval
A beating feeling interval estimation unit that estimates a beating feeling interval so as to be smaller than an estimated value of the beating feeling interval;
Using the speed of the subject's utterance, the estimated value of the beating feeling intensity corresponding to the section in which the speed of the subject's utterance is the first value is the second value where the speed of the subject's utterance is smaller than the first value. Corresponds to the interval
Including at least one of the pulsation feeling intensity estimation unit for estimating the pulsation feeling intensity so as to be larger than the estimated value of the pulsation feeling intensity.
Distribution system. A distribution system including a distribution server and a heartbeat information generating device,
The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
The distribution server estimates at least one of a beating feeling interval and a beating feeling intensity using an audio signal including a speech of a subject collected by a microphone, and the estimated value of the beating feeling interval and the beating feeling Distributing at least one of the estimated intensity value to the beating sensation information generating device,
The beating feeling information generating device
A pulsation feeling information presentation unit that synthesizes and presents pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity received from the distribution server;
The heartbeat information presented by the heartbeat information presentation unit is :
The sound for one heart beat prepared in advance is a sound that is repeated with the estimated value of the beating feeling interval, and / or the sound volume is increased as the estimated value of the beating feeling intensity increases. which is the difference between the sound,
Distribution system. A distribution system including a distribution server and a heartbeat information generating device,
The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
The distribution server estimates at least one of a beating feeling interval and a beating feeling intensity using an audio signal including a speech of a subject collected by a microphone, and the estimated value of the beating feeling interval and the beating feeling Distributing at least one of the estimated intensity value to the beating sensation information generating device,
The beating feeling information generating device
A pulsation feeling information presentation unit that synthesizes and presents pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity received from the distribution server;
The heartbeat information presented by the heartbeat information presentation unit is:
A waveform of a heartbeat prepared in advance is an image of a waveform repeated with the estimated value of the beating feeling interval, and / or the larger the estimated value of the beating feeling intensity, the larger the amplitude of the waveform. It is a video of the waveform synthesized in the
Distribution system. The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
A pulsation feeling information estimation step for estimating at least one of the pulsation feeling interval and the pulsation feeling intensity from the speech of the subject person;
Including a pulsation feeling information synthesis step of synthesizing pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity,
The beating feeling information estimation step includes:
Using the speed of the subject's utterance, the estimated value of the beating feeling interval corresponding to the section in which the speed of the subject's utterance is the first value is the second value in which the speed of the subject's utterance is smaller than the first value. A beating feeling interval estimation step for estimating a beating feeling interval so as to be smaller than an estimated value of the beating feeling interval corresponding to a section,
Using the speed of the subject's utterance, the estimated value of the beating feeling intensity corresponding to the section in which the speed of the subject's utterance is the first value is the second value where the speed of the subject's utterance is smaller than the first value. Including at least one of the pulsation feeling intensity estimation steps for estimating the pulsation feeling intensity so as to be larger than the estimated value of the pulsation feeling intensity corresponding to the section.
Heartbeat information generation method. The sensation that the animal feels for the heartbeat is the beating sensation, the interval between beating sensations is the beating sensation interval, the strength of the beating sensation is the beating feeling intensity, and the information for presenting the beating sensation is the beating sensation Information and
A pulsation feeling information estimation step for estimating at least one of the pulsation feeling interval and the pulsation feeling intensity from the speech of the subject person;
Including a pulsation feeling information synthesis step of synthesizing pulsation feeling information from at least one of the estimated value of the pulsation feeling interval and the estimated value of the pulsation feeling intensity,
The beating feeling information estimation step includes:
With 0.5 <α <1, and using the subject's utterance speed, the estimated value of the beating sensation interval corresponding to the section in which the subject's utterance speed is the first value is the target utterance speed. The first beating feeling interval is estimated to be smaller than the estimated value of the beating feeling interval corresponding to the interval that is the second value smaller than the value, and the subject's utterance is used by using the basic frequency of the subject's utterance. The estimated value of the beating feeling interval corresponding to the section in which the estimated value of the beating feeling interval corresponding to the section in which the fundamental frequency of the subject is the first value is the second value in which the fundamental frequency of the subject's utterance is smaller than the first value. The second pulsation feeling interval is estimated to be smaller than the first pulsation feeling interval estimated value BD1 and the second pulsation feeling interval estimation value BD2.
A beating sensation interval estimation step,
Using 0.5 <β <1 and the subject's utterance speed, the estimated value of the beating sensation intensity corresponding to the section where the subject's utterance speed is the first value is the subject's utterance speed is the first. The first pulsation feeling intensity is estimated to be larger than the estimated value of the pulsation feeling intensity corresponding to the second value interval that is smaller than the value, and the target person's utterance is used by using the basic frequency of the utterance of the target person. The estimated value of the beating feeling intensity corresponding to the section in which the fundamental frequency of the subject's utterance is the second value smaller than the first value is the estimated value of the beating feeling intensity corresponding to the section in which the basic frequency is the first value. The second pulsation feeling intensity is estimated to be larger than the first pulsation feeling intensity estimated value BS1 and the second pulsation feeling intensity estimated value BS2.
Including a beating sensation intensity estimation step
Heartbeat information generation method.   A program for causing a computer to function as the beating sensation information generation device according to any one of claims 1 to 6.