JP5594152B2 - NAM conversation support system and NAM conversation support method - Google Patents

Description

  The present invention relates to a NAM conversation support system and a NAM conversation support method for supporting NAM conversation by a plurality of persons using NAM (Non Audible Murmur).

  Various conversation systems using NAM are currently being researched. NAM is a non-audible tweet, and is unvoiced speech that cannot be heard by others within a radius of 1 m. The NAM is performed using, for example, breathing exhalation. Unvoiced speech is a sound that passes through the vocal cords located above the larynx without shaking when exhalation from the lungs passes through the larynx when adjusting speech.

For example, as a voice by NAM, a microphone that collects a vibration sound that is not accompanied by a regular vibration of the vocal cords and is accompanied by a very small amount of respiration and that is conducted through a soft tissue of the body such as a speaker's meat, and a communication using this microphone Interface systems are known.
Since the voice using NAM is an unvoiced sound that does not use a vocal cord as described above, it is difficult to identify a speaker using a faultman that is usually used for voiced sound. Therefore, in the communication system using the NAM, when there are a plurality of speakers, it is difficult to identify the speakers other than visually.

  On the other hand, there is known a communication system that can easily identify a user who is speaking and a user who is not speaking even when a plurality of users are speaking at the same time. The communication system includes a display that specifies a speaker and a non-speaker using control information including identification information for identifying a speaker and displays information related to each speaker and non-speaker in different formats.

WO2004 / 021738 A1 JP 2007-288255 A

In the above communication system using NAM, when three or more people have a conversation with NAM and two or more people speak, NAM conversation participants who are listeners can talk to each other when the speakers are close to each other. Since the faces can be matched, it is possible to identify who is the speaker. If they are not close to each other, for example, about 100 m away, it is impossible to identify who is generating only by the NAM unvoiced speech.
On the other hand, when the communication system is used, since a display for displaying information on speakers and non-speakers in different formats is provided, it is difficult for a plurality of NAM conversation participants to easily use the communication system anywhere.

  Therefore, according to the present invention, when supporting NAM conversation by a plurality of persons using NAM, each NAM conversation participant can easily identify the NAM speaker by listening to the NAM conversation by the NAM speaker. An object is to provide a support system and a NAM conversation support method.

One aspect of the present invention is a NAM conversation support system that supports NAM conversation by a plurality of persons using NAM (Non Audible Murmur). The system
A measurement unit that acquires position information indicating the current position of each NAM conversation participant and information on a direction in which each NAM conversation participant is currently facing; a voice input / output device that each NAM conversation participant uses for NAM conversation; A mobile terminal device held by each NAM conversation participant, comprising: a communication unit that performs communication of the position information and the orientation information and communication of an audio signal during NAM conversation;
Based on the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker as seen from each NAM conversation participant specified using the position information and the orientation information received from the mobile terminal device, The NAM speaker's assigned direction and the NAM speaker's assigned fundamental frequency for each NAM conversation participant are determined, and the NAM silent voice signal received from the portable terminal device based on the assigned direction and the assigned fundamental frequency A conversation management device comprising: a signal processing unit for processing; and a communication unit for transmitting the processed silent voice signal to the portable terminal device.
At this time, when the signal processing unit determines the allocation direction and the allocation fundamental frequency for the NAM speaker, a plurality of blocks each having a plurality of preset division directions and a plurality of preset division frequencies as a set. Therefore, one block is allocated to the NAM speaker so that a block overlapping with another NAM speaker is not allocated.
Alternatively, when the signal processing unit determines the assigned direction and the assigned fundamental frequency for the NAM speaker, the signal processing unit uses the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker to create one block. Assign to the NAM speaker.
Alternatively, the signal processing unit, for NAM speakers located within a predetermined distance range when viewed from each NAM conversation participant, assigns the current orientation of the NAM speaker and the voice fundamental frequency to the assigned orientation and Set as the assigned fundamental frequency.

Another aspect of the present invention is a NAM conversation support method performed by a conversation management device that supports NAM conversation by a plurality of persons using NAM (Non Audible Murmur). In this method,
Receiving position information indicating the current position of each NAM conversation participant and information on the direction in which each NAM conversation participant is currently facing from a mobile terminal device held by each NAM conversation participant;
Each NAM conversation participant is based on the current direction of the NAM speaker and the voice fundamental frequency of the NAM speaker as viewed from each NAM conversation participant identified using the received position information and the orientation information. Defining the NAM speaker's allocation direction and the NAM speaker's allocation fundamental frequency for
A NAM unvoiced voice signal is processed based on the assigned direction and the assigned fundamental frequency, and the processed unvoiced voice signal is transmitted to each NAM conversation participant.
When determining the allocation direction of the NAM speaker and the allocation fundamental frequency of the NAM speaker, another NAM message is determined from a plurality of blocks each having a set of a plurality of preset division directions and a plurality of preset division frequencies. One block is allocated to the NAM speaker so that a block overlapping with the speaker is not allocated. Alternatively, a block is assigned to the NAM speaker using the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker. Alternatively, for a NAM speaker located within a predetermined distance range when viewed from each NAM conversation participant, the current direction and the voice fundamental frequency of the NAM speaker are set as the assigned direction and the assigned fundamental frequency. To do.

  In the NAM conversation support system and the NAM conversation support method according to the above-described aspect, when supporting a NAM conversation by a plurality of persons using the NAM, each NAM conversation participant listens to the NAM conversation by the NAM speaker, thereby It can be easily identified.

It is a figure explaining the whole outline of the NAM conversation support system of this embodiment. It is a figure which shows the structure of the portable terminal device and NAM management server in this embodiment. It is a figure explaining an example of allocation of the azimuth | direction and fundamental frequency which the frequency and azimuth | direction allocation part of the NAM conversation management server shown in FIG. 2 performs. It is a figure explaining the other example of allocation of the azimuth | direction and fundamental frequency which the frequency and azimuth | direction allocation part of the NAM conversation management server shown in FIG. 2 performs. It is a figure explaining the other example of allocation of the azimuth | direction and fundamental frequency which the frequency and azimuth | direction allocation part of the NAM conversation management server shown in FIG. 2 performs. It is a figure explaining the other example of allocation of the azimuth | direction and fundamental frequency which the frequency and azimuth | direction allocation part of the NAM conversation management server shown in FIG. 2 performs. It is a figure explaining the state where a NAM speaker approaches a NAM conversation participant during NAM conversation using the NAM conversation support system of this embodiment. It is a figure explaining the state where a NAM speaker moves around a NAM conversation participant during NAM conversation using the NAM conversation support system of this embodiment. It is a figure which shows the flow of an example of the setting of the allocation fundamental frequency and allocation direction in this embodiment. It is a figure which shows the flow of the other example of the setting of the allocation fundamental frequency and allocation direction in this embodiment. It is a figure which shows the flow of the further another example of the setting of the allocation fundamental frequency and allocation direction in this embodiment.

Hereinafter, the NAM conversation support system and NAM conversation support method of the present invention will be described.
FIG. 1 is a diagram illustrating an overall outline of a NAM conversation support system 10 of the present embodiment.

(NAM conversation support system)
The NAM conversation support system 10 shown in FIG. 1 shows an example in which NAM conversation participants A to E face NAM conversations in different directions. Any of the NAM conversation participants A to E can become a NAM speaker if he / she speaks at the NAM.
The NAM conversation participants A to E carry the portable terminal devices 12a to 12e. Each of the mobile terminal devices 12a to 12e has a headset in which a microphone and headphones are integrated as a voice input / output device, and the NAM conversation participants A to E wear the headset. The portable terminal devices 12a to 12e acquire position information indicating the current positions of the NAM conversation participants A to E and information on the direction in which the NAM conversation participants A to E are currently facing by using sensors, and perform NAM conversation. It transmits to the management server 14. The NAM conversation management server 14 identifies the current direction of the NAM speaker viewed from each of the NAM conversation participants A to E using the received position information and direction information, and identifies the current direction of the identified NAM speaker. And an allocation fundamental frequency based on the voice fundamental frequency of the NAM speaker. Using the assigned direction and the assigned fundamental frequency, the NAM speaker's silent voice signal is processed and transmitted to each NAM conversation participant A to E. The portable terminal devices 12a to 12e and the NAM conversation management server 14 use multi-channel signals such as stereo signals as audio signals.

  For this reason, even in the case of unvoiced speech that is difficult to identify in NAM conversation, the fundamental frequency of the unvoiced speech is converted in accordance with the assigned fundamental frequency, and the unvoiced speech signal that is a multi-channel signal is matched to the assigned direction. Since the signal processing is performed, each of the NAM conversation participants A to E can identify the NAM speaker from the fundamental frequency of the unvoiced speech and the direction in which the NAM speaker can hear the NAM. Hereinafter, the mobile terminal devices 12a to 12e and the NAM conversation management server 14 will be described.

(Mobile terminal device)
FIG. 2 is a diagram showing the configuration of the mobile terminal device 12a and the NAM management server 14. Since all the portable terminal devices 12a to 12e have the same structure, the portable terminal device 12a will be described below as a representative of the portable terminal devices 12a to 12e. In addition, when the mobile terminal devices 12a to 12e are generically described, they are referred to as the mobile terminal device 12.
The mobile terminal device 12 includes a headset 16 that is a voice input / output device, a measurement unit 18, a Lombard detection unit 20, and a communication unit 22.
The headset 16 includes a headphone 16a capable of listening to NAM conversation through a speaker, a microphone 16b including a microphone for voiceless voice capable of collecting voiceless voice of the NAM and a voiced voice microphone capable of collecting normal voiced voice. Have
The microphone for silent voice is exemplified by a microphone described in WO2004 / 021738 A1, for example. Specifically, a stethoscope-type microphone is attached as a microphone for unvoiced speech on the body surface just below the mastoid process at the rear lower part of the auricle of the skull. The stethoscope-type microphone includes a diaphragm having an adhesive surface that adheres to the body surface, and a condenser microphone fixed to the diaphragm via a suction disk.
When a NAM speaker utters a NAM without using the regular vibration of the vocal cords, the NAM is tuned by the speech movement of the articulatory organs such as the tongue, lips, chin, and soft palate and also vibrates in the body's soft tissue such as meat. As it reaches the body surface just below the mastoid process. The microphone for silent voice picks up this vibration sound.
The headphone 16a reproduces a whisper sound having a fundamental frequency such as approximately 100 to 300 Hz in addition to voiced sound.
The voiced voice signal and the voiceless voice signal obtained by the microphone 16 b are sent to the communication unit 22. The headphone 16 a converts the NAM conversation signal transmitted from the NAM management server 14 into sound and reproduces it.

The measurement unit 18 includes a GPS (Global Positioning System) position sensor 18a that acquires position information indicating the current position of each NAM conversation participant A to E, and a direction in which each NAM conversation participant A to E is currently facing. It has a direction sensor 18b for acquiring information. The position information and orientation information obtained by the measurement unit 18 are sent to the communication unit 22 and transmitted to the NAM management server 14.
The location information and orientation information of each NAM conversation participant A to E is transmitted to the NAM management server 14 in the NAM management server 14 in the current direction of the NAM speaker viewed from each NAM conversation participant A to E. This is because

  The Lombard detection unit 20 detects whether or not voiced sound is included in the voice emitted from the microphone 16b. For example, the Lombard detection unit 20 is voiced to the voice of a NAM speaker by examining the power value of a voice signal in a predetermined frequency band (frequency band corresponding to formant) that is included in voiced voice but not included in unvoiced voice. Detect whether audio is included. The detection result is sent to the communication unit 22.

  The communication unit 22 transmits signals and information transmitted from the headset 16, the measurement unit 18, and the Lombard detection unit 20 to the NAM management server 14 by wireless communication, and also transmits signals transmitted from the NAM management server 14. Receive. FIG. 2 functionally shows the communication unit 22. That is, the communication unit 22 includes a voiced voice transmission unit 22a, a NAM transmission unit 22b, a Lombard transmission unit 22c, a position / orientation transmission unit 22d, and a voice signal reception unit 22e.

  The voiced voice transmission unit 22a generates a transmission signal so as to transmit the voiced voice signal obtained by the voiced voice microphone of the microphone 16b. The NAM transmission unit 22b generates a transmission signal so as to transmit an unvoiced voice signal obtained by the unvoiced voice microphone of the microphone 16b. The Lombard transmission unit 22c generates a transmission signal so as to transmit information on the detection result obtained by the Lombard detection unit 20. The position / azimuth transmission unit 22d generates a transmission signal so as to transmit the information obtained by the measurement unit 18. The audio signal receiving unit 22e receives the transmission signal transmitted from the NAM management server 14 and generates an audio signal. The communication unit 22 includes a transmitter and a receiver not shown. A transmitter (not shown) transmits the transmission signal generated by each transmitter to the NAM conversation management server 14. A receiver (not shown) receives a signal transmitted from the NAM conversation management server 14.

(NAM conversation management server)
The NAM conversation management server 14 processes various information, voiced voice signals, and unvoiced voice signals transmitted from the mobile terminal device 12 and transmits them to the mobile terminal device 12.
The NAM conversation management server 14 includes a communication unit 30 and a signal processing unit 32. The NAM conversation management server 14 is a computer having a CPU, RAM, ROM, and storage unit (not shown), and the CPU, RAM, ROM, and storage unit are connected to each other via a bus.
Of the signal processing unit 32, a fundamental frequency calculation unit 32a, a frequency / orientation allocation unit 32c, a whispering voice conversion unit 32d, and a voice mixing unit 32e, which will be described later, read and execute programs stored in the ROM or the storage unit. It is a software module generated by doing. In other words, a CPU (not shown) has the functions of a fundamental frequency calculation unit 32a, a frequency / orientation assignment unit 32c, a whispering voice conversion unit 32d, and an audio mixing unit 32e.
The communication unit 30 includes a voiced voice receiving unit 30a, a NAM receiving unit 30b, a Lombard receiving unit 30c, a position / orientation receiving unit 30d, and a voice signal transmitting unit 30e.

  The voiced voice receiving unit 30a extracts a voiced voice signal from the received transmission signal. The NAM receiving unit 30b extracts a silent voice signal from the received transmission signal. The Lombard receiving unit 30c extracts information on the detection result obtained by the Lombard detecting unit 20 from the transmission signal. The position / orientation receiving unit 30d extracts information obtained by the measuring unit 18 from the transmission signal. The audio signal transmission unit 30e generates a transmission signal so as to transmit the audio signal transmitted from each of the mobile terminal devices 12a to 12e and subjected to signal processing and mixing to each of the mobile terminal devices 12a to 12e. The communication unit 30 includes a transmitter and a receiver not shown. A receiver (not shown) receives signals transmitted from the mobile terminal devices 12a to 12e. A transmitter (not shown) transmits the transmission signal generated by the audio signal transmission unit 30e to each of the mobile terminal devices 12a to 12e.

The signal processing unit 32 includes a fundamental frequency calculation unit 32a, a database 32b, a frequency / orientation allocation unit 32c, a whispering voice conversion unit 32d, and an audio mixing unit 32e.
The fundamental frequency calculation unit 32a calculates the fundamental frequency of the voice of the NAM speaker by performing frequency analysis on the voiced speech signal generated by the voiced speech reception unit 30a. Voiced speech has a fundamental frequency and a frequency harmonized with a fundamental frequency such as overtones. The fundamental frequency of voiced speech matches the speech fundamental frequency of unvoiced speech of NAM. That is, the fundamental frequency of the NAM can be obtained by analyzing the frequency of the voice of the NAM speaker in advance. For this reason, the fundamental frequency calculation unit 32 calculates in advance the fundamental frequency of the voice of each NAM conversation participant participating in the NAM conversation, that is, the fundamental speech frequency, and this fundamental speech frequency is used as the fundamental speech frequency at the time of NAM speech. It records in the database 32b with the identification information of the portable terminal device 12. FIG. A part of the storage unit (not shown) of the computer that is the NAM conversation management server 14 is used as the database 32b.

When the NAM receiving unit 30c receives the NAM silent voice signal, the frequency / orientation allocating unit 32c extracts the NAM speaker's voice fundamental frequency from the database 32b. At this time, the frequency / orientation allocating unit 32c further uses the position information and orientation information of each NAM conversation participant A to E received from the mobile terminal device 12 to view the NAM viewed from each NAM conversation participant A to E. Identify the current orientation of each speaker. Since the frequency / orientation allocating unit 32c obtains position information and orientation information of each NAM conversation participant A to E, the current direction of each speaker can be easily specified. Further, since the frequency / orientation allocating unit 32c knows from which mobile terminal device the unvoiced voice by the received NAM is transmitted, the voice basic frequency recorded in the database 32b is determined from the identification information of the mobile terminal device. It can be taken out.
At this time, if an allocation direction and an allocation fundamental frequency, which will be described later, are already set for the NAM speaker, and if the allocation direction and the allocation fundamental frequency are not changed, the following processing is not performed, and the whispering voice conversion unit 32d and In the audio mixing unit 32e, the assigned allocation direction and the assigned fundamental frequency are used as they are. On the other hand, when the allocation azimuth and the allocation fundamental frequency are not set, or when the allocation azimuth and the allocation fundamental frequency are changed, the frequency / azimuth allocation unit 32c sets the allocation azimuth and the allocation fundamental frequency described later. The case where the assigned direction and the assigned fundamental frequency are not set is a case where the user speaks for the first time after participating in the NAM conversation, or who participates in the NAM conversation, but there are few utterances within a certain period, and the assigned direction and assigned fundamental frequency. Is a case where the assigned orientation and the assigned fundamental frequency are changed, for example, when the NAM speaker or NAM conversation participant moves during the NAM conversation and the orientation changes.

  When setting the assigned direction and the assigned fundamental frequency, or when changing the assigned direction and the assigned fundamental frequency, the frequency / orientation assigning unit 32c determines the current direction identified as the voice fundamental frequency. The NAM speaker allocation direction and NAM speaker allocation fundamental frequency for each NAM conversation participant are determined based on the above. Specifically, the frequency / azimuth allocating unit 32c divides a divided azimuth obtained by dividing a 360 degree azimuth by a preset number and a predetermined frequency range, for example, a range of 100 to 160 Hz, by a preset number. One block is assigned to a NAM speaker so that a block overlapping between NAM speakers is not assigned from a plurality of blocks having a set of division frequencies. At that time, the block is allocated to the NAM speaker using the current direction of the NAM speaker and the voice fundamental frequency of the NAM speaker. The setting of the allocation direction and the allocation fundamental frequency will be described later.

The whispering voice conversion unit 32d converts the frequency of the voice signal so that the assigned fundamental frequency becomes the NAM voice fundamental frequency, and further increases the signal level so that the NAM that is a whisper becomes voice equivalent to the whispering voice. A known conversion process is used to convert the frequency of the audio signal. Examples of the conversion processing include STRIGHT by Kawara et al. (Kawahara et al., Shingaku Techniques, EA 96-28, 1996).
The voice mixing unit 32e uses the NAM speaker's assigned orientation determined by the frequency / orientation assigning unit 32c so that the NAM can hear unvoiced speech from the direction in which the NAM substantially matches the vision as seen from each NAM conversation participant. Mixing processing of NAM silent voice signal. Since the audio signal is a multi-channel signal such as a stereo signal, the audio mixing unit 32e processes the audio signal according to the assigned direction as viewed from each NAM conversation participant. Furthermore, in the audio mixing unit 32e, when a plurality of NAM speakers speak at the same time, a plurality of audio signals are mixed. The audio mixing unit 32e sends the mixed audio signal to the audio signal transmission unit 30e.

A NAM speaker utters voiced speech in some cases during a NAM conversation. In this case, the Lombard receiving unit 30c obtains information on the detection result performed by the Lombard detecting unit 20, and this information is sent to the audio mixing unit 32e. When the voice mixing unit 32e obtains a detection result indicating that the voice of the NAM speaker is voiced voice, the voice mixing unit 32e switches from the voiceless voice mode supporting voiceless voice conversation by the NAM to the voiced voice mode supporting voiced voice conversation. Specifically, the voice mixing unit 32e blocks another unvoiced voice signal during the NAM conversation, and sends the voiced voice signal sent from the voiced voice reception unit 30a to each portable terminal device 12a to 12a through the voice signal transmission unit 30e. 12e. That is, when a microphone 16b picks up voiced speech when a certain NAM speaker speaks, the NAM conversation management server 14 cuts off the voiceless voice signal during the NAM conversation and carries the voiced voice signal through the communication device 30. It transmits to terminal device 12a-12e.
The voiced voice mode automatically returns to the voiceless voice mode when the NAM speaker does not detect voiced voice for a certain period of time, for example, 10 seconds.
The NAM conversation support system according to the present embodiment automatically switches from the voiceless voice mode to the voiced voice mode when the NAM conversation participant utters voiced voice, but does not have the voiced voice mode but only the voiceless voice mode. Support may be provided.

(Assignment of frequency and direction)
3 to 5 are diagrams for explaining an example of the direction and fundamental frequency assignment performed by the frequency / orientation assignment unit 32c described above.
In the example shown in FIG. 3, the frequency / orientation allocating unit 32c divides the 360 degree azimuth into eight parts, and each 45 degrees north, northeast, east, southeast, south, etc. as viewed from each NAM conversation participant A to E. Eight division directions are defined. In addition, the frequency / azimuth allocation unit 32c divides the frequency range of 100 to 160 Hz into six as the allocation fundamental frequency, and is a band of six division frequencies of 150 Hz or more, 140 to 150 Hz, 130 to 140 Hz, ..., less than 110 Hz. Determine. FIG. 3 is an example of assignment when the NAM conversation participant is a male. FIG. 4 is an example of assignment of division directions and division frequencies when the NAM conversation participant is a woman. In the case of a woman, the assigned fundamental frequency is set to a band of seven divided frequencies of less than 220 Hz, 220 to 230 Hz, 230 to 240 Hz,. When setting a basic frequency to be assigned to each NAM conversation participant, a value obtained by adding ± 10 Hz, ± 20 Hz,..., To the basic frequency of each NAM conversation participant is assigned as a divided frequency. For example, when the basic frequency of the NAM conversation participant is 132 Hz and the division frequency to be assigned is a frequency in a band of 140 to 150 Hz, it is 142 Hz. Similarly, in a band of 150 Hz or more and less than 110 Hz, the frequency is set to 152 Hz or 102 Hz. Also in the direction, for example, values such as ± 45 °, ± 90 °,...

  In the example shown in FIG. 3, the frequency / orientation allocating unit 32 c does not allocate overlapping blocks among NAM speakers from 48 (= 8 × 6) blocks in which the division direction and the division frequency are set. Thus, one block is allocated to each NAM speaker. Therefore, in the example shown in FIGS. 3 and 4, the same assigned fundamental frequency can be assigned to eight NAM speakers under conditions where the assigned directions are different. In that case, a block is allocated to each NAM speaker using the current direction of each NAM speaker and the voice fundamental frequency of each NAM speaker.

FIG. 5 is a diagram for explaining assignment using the example shown in FIG. Allocation is performed using the priority of allocation. Specifically, the frequency / orientation allocating unit 32c sets priorities in the order of NAM speakers having a large number of times or the number of times of speech in the past fixed time, for example, the past 3 minutes, and according to the priorities, To determine the assigned direction and assigned fundamental frequency. Prioritization of assignment may be determined by the number or time of the above-mentioned utterances, the priorities of the NAM conversation participants may be determined by a meeting before the start of the NAM conversation, or may be made by the NAM immediately after the start of the NAM conversation. Priorities may be set in order.
In the example shown in FIG. 5, one person is assigned to 140 to 150 Hz, one person is assigned to 130 to 140 Hz, two persons are assigned to 120 to 130 Hz, and eight persons are assigned to 110 to 120 Hz. Such allocation information is preferentially examined when attempting to allocate blocks to NAM speakers. That is, when allocating a block to a NAM speaker, the frequency / orientation allocating unit 32c examines the “number of allocations” in the division frequency column corresponding to the voice fundamental frequency of the NAM speaker shown in FIG. If it is less than 8, it is checked whether or not a block corresponding to the current direction of the NAM speaker is already allocated in the division frequency column. If this block is not assigned, it is assigned to the NAM speaker. On the other hand, when the block is already assigned to another NAM speaker, the frequency / orientation assigning unit 32c searches for an unassigned adjacent block.

For example, the NAM speaker A who has a voice fundamental frequency in the range of 120 to 130 Hz and is currently located in the southeast direction as seen from a certain NAM conversation participant speaks as shown in FIG. Assume that a state is assigned to a block X having a division frequency and a division direction in the southeast. At this time, when NAM speaker B who has a voice fundamental frequency of 120 to 130 Hz and is located in the southeast direction as viewed from the NAM conversation participant speaks, since NAM speaker A is already assigned to block X. The frequency / orientation assigning unit 32c assigns the blocks X _{1 to} Y ₄ having adjacent divided frequencies or adjacent divided directions adjacent to each other in the divided frequency or divided direction of the block X to be assigned. Since the NAM conversation participant can easily identify the NAM speaker by the NAM speaker's voice fundamental frequency, the frequency / orientation allocation unit 32c gives priority to not changing the voice fundamental frequency, and the orientation is adjacent. It is preferable to change the division direction. That is, in the example of FIG. 3, it is preferable that the frequency / orientation assigning unit 32c assigns to the block Y ₁ or Y ₄ . When blocks Y ₁ and Y ₄ are already assigned to another NAM speaker, one of blocks Y ₂ and Y ₃ is assigned without changing orientation. If all of the blocks Y _{1 to} Y ₄ are already assigned, the priority is lower than that of the NAM speaker B, and the four NAM speakers assigned with the _four blocks Y _{1 to} Y ₄ are assigned. Of these, the block assigned to the NAM speaker with the lowest priority is forcibly assigned. At this time, if the adjacent block of the block for which the assignment of the NAM speaker having a low priority is canceled can be assigned, the frequency / orientation assigning unit 32c assigns the adjacent block to the NAM speaker for which the assignment has been canceled. Also good. The example shown in FIG. 5 shows that the NAM speaker C assigned to the block Z is canceled and assigned to the NAM speaker B ₄ .

  Further, the frequency / orientation assigning unit 32c releases the block assigned to the NAM speaker G who has not made a speech for a certain past time, for example, 10 minutes. However, in this case, the frequency / orientation assigning unit 32c may maintain the “assignment number” of the divided frequencies as eight. This is because when the NAM speaker G who has not made a speech suddenly resumes the speech, the released block can be allocated. At this time, when the NAM speaker H whose priority is higher than that of the NAM speaker G speaks, and the frequency / orientation allocation unit 32c tries to allocate the block allocated to the NAM speaker G to the NAM speaker H, As shown in block α in FIG. 5, block α may be assigned to NAM speaker H instead of NAM speaker G. At this time, if the adjacent block β adjacent to the block α can be assigned, the frequency / orientation assigning unit 32c may assign the adjacent block β to the NAM speaker G.

As shown in FIG. 6, when assigning a block to the NAM speaker D, the frequency / orientation assigning unit 32c does not have the same orientation as the NAM speaker A with the same assigned fundamental frequency, but with the NAM speaker C with the same assigned orientation. One block is determined so that the assigned fundamental frequencies are not the same.
Such block allocation is performed not only for NAM speakers to which no block is allocated, but also when the NAM speaker moves in different directions. In this case, the set allocation direction is changed to a new allocation direction.

The setting of the allocation fundamental frequency and the allocation direction as shown in FIGS. 3 to 6 is preferably performed when the NAM speaker is located far from each NAM conversation participant. To be located far away means that NAM speakers are located apart from each other by a predetermined distance or more when viewed from each NAM conversation participant.
In addition, when the NAM speaker is located near to each NAM conversation participant as viewed from each NAM conversation participant, the frequency / orientation allocating unit 32c allocates the current direction of the NAM speaker and the voice fundamental frequency to the NAM speaker. It is preferable to set the direction and the assigned fundamental frequency. “Nearly located” means that a NAM speaker is located within a predetermined distance range as viewed from each NAM conversation participant. In this case, the assigned fundamental frequency and assigned direction may overlap with other NAM speakers. Even if the assigned fundamental frequency and orientation are overlapped with other NAM speakers, the NAM speakers are located close to each other as viewed from each NAM conversation participant, so the NAM speaker's mouth movements and facial expressions can be visually observed. Thus, the NAM speaker can be sufficiently identified.

FIG. 7 shows a state in which the NAM speaker A moves and approaches the NAM conversation participant P. The method of determining the allocation fundamental frequency and the allocation direction as shown in FIG. 5 is used in FIG. 7 when the NAM speaker A is located in a range separated from the NAM conversation participant P by a predetermined distance L ₁ or more. . On the other hand, when the NAM speaker A is located within a predetermined distance L ₂ from the NAM conversation participant P, the frequency / orientation assigning unit 32c determines the current orientation of the NAM speaker A as the assigned orientation, The voice fundamental frequency of person A is assigned and determined as the fundamental frequency. Furthermore, the frequency and direction assignment unit 32c, in between the distance L ₁ and the distance L _2, modifying the allocation orientation and allocated fundamental frequency in accordance with the distance between the NAM speaker A and NA conversation participants P It is preferable to do.
For example, when the distance from the NAM conversation participant P to the NAM speaker A is x, the assigned fundamental frequency is f (x) and the assigned direction θ (x) is expressed by the following equations (1) and (2). expressed.
F in the formula (1) is a voice fundamental frequency, and F ′ is an assigned fundamental frequency determined by the method shown in FIGS. In equation (2), ψ is the current orientation, and ψ ′ is the assigned orientation determined by the method shown in FIGS. The distance L ₁ is 100 m, for example, and the distance L ₂ is 10 m, for example.

  At this time, the frequency / orientation assigning unit 32c keeps the assigned orientation or the assigned fundamental frequency close to the current orientation of the NAM speaker A as viewed from the NAM conversation participant P or the voice fundamental frequency of the NAM speaker A. It is preferable that the correction be performed smoothly over time T. For example, when the NAM speaker moves and arrives at the destination, the assigned fundamental frequency at the destination is F ′, the assigned fundamental frequency at the source is F, the assigned direction at the destination is ψ ′, and the source When the assigned direction is ψ, the assigned fundamental frequency f (t) and the assigned direction θ (t) are expressed according to the following formulas (3) and (4). T in the equations (3) and (4) is, for example, 3 seconds. When the NAM conversation participant P moves across a plurality of blocks, the frequency / orientation assigning unit 32c may set the assigning direction and the assigning frequency as F ′ and ψ ′ for each passing block.

  Also, as shown in FIG. 8, when the NAM speaker B is moving around the NAM conversation participant P, the allocation direction depends on the current direction of the NAM speaker B viewed from the NAM conversation participant P. It is preferable to make corrections so as to change smoothly over a certain period of time. At this time, as described above, when the block to be allocated changes due to the movement of the NAM speaker B and the changed previous block has already been allocated to another NAM speaker, the frequency / direction allocation unit 32c changes An adjacent block adjacent to the previous block may be assigned. Thus, when the NAM speaker B moves, the NAM conversation participant P can hear the movement of the NAM speaker B by voice.

(NAM conversation support method)
In the NAM conversation support system 10, the voices of the NAM conversation participants A to E before the start of the NAM conversation are collected. The NAM conversation management server 14 uses the voiced voices of the NAM conversation participants A to E obtained via the voiced voice receiver 30a to calculate the fundamental frequency of voiced voice at the fundamental frequency calculator 32, which is used as the voice fundamental frequency. Recorded in the database 32b.
Thereafter, the NAM conversation is started. During the NAM conversation, the NAM receiving unit 30b displays the position information indicating the current position of each NAM conversation participant and the information on the direction in which each NAM conversation participant A to E is currently facing each NAM conversation participant A to E. Are continuously received from the mobile terminal devices 12a to 12e held by the device.
The frequency / azimuth allocation unit 32c records the current direction of each NAM speaker as viewed from each NAM conversation participant A to E specified using the received position information and direction information, and records each NAM speaker. The allocation direction of each NAM speaker for each NAM conversation participant A to E and the allocation fundamental frequency of each NAM speaker are determined based on the voice basic frequency. The allocation frequency and allocation direction as shown in FIGS. 3 to 5 are set for NAM speakers located in a range farther than the distance L ₁ as shown in FIG. 7 when viewed from each NAM conversation participant. . When the allocation direction and the allocation fundamental frequency are already set for the NAM speaker and are not changed, the frequency / direction allocation unit 32c does not perform allocation. For NAM speakers located within a range closer to the distance L ₂ as shown in FIG. 7 when viewed from each NAM conversation participant, the current direction of the NAM speaker and the voice fundamental frequency of the NAM speaker are assigned directions and assigned basics. Set as frequency.
The whispering voice conversion unit 32d and the audio mixing unit 32e perform signal processing (increase in signal level, conversion of basic frequency, and mixing) of the NAM unvoiced audio signal based on the assigned direction and the assigned fundamental frequency. The voice signal transmitting unit 30e transmits the processed silent voice signal to each of the NAM conversation participants A to E.

  As described above, in the NAM conversation support method, the allocation direction of each NAM speaker for each NAM conversation participant A to E based on the current direction of each NAM speaker and the voice fundamental frequency of each NAM speaker. The assigned fundamental frequency of each NAM speaker is determined, and signal processing of the NAM unvoiced speech signal is performed based on the assigned direction and assigned fundamental frequency. Therefore, each NAM conversation participant can identify the NAM speaker by listening to the whispering voice obtained by converting the NAM by the NAM speaker. That is, each NAM conversation participant can know which person in which direction issued the NAM.

FIGS. 9 to 11 show flows in various cases of setting the allocation fundamental frequency and the allocation direction when the NAM speaker A speaks to the NAM conversation participant using the NAM.
In the example shown in FIG. 9, the NAM speaker A speaks to the NAM conversation participant with the NAM. At this time, the frequency / orientation allocating unit 32c newly sets a basic frequency and an allocation direction to be assigned to the NAM speaker A by the assignment method shown in FIGS. 3 to 6 for the NAM speaker A (step S10). In this state, when the NAM speaker A moves to a different direction (Yes in step S20), the frequency / direction assigning unit 32c maintains the assigned fundamental frequency and changes the assigned direction (step S30). Specifically, the frequency / orientation allocating unit 32c sets a divided azimuth that is close to the actual moving destination azimuth as the allocated azimuth. When the NAM speaker A does not move to a different direction (No in step S20), the frequency / direction assigning unit 32c maintains the assigned fundamental frequency and the assigned direction (step S40). Thus, the NAM conversation is continued.

In the example shown in FIG. 10, the NAM speaker A speaks to the NAM conversation participant using the NAM. At this time, the frequency / orientation allocating unit 32c newly sets a basic frequency and an allocation direction to be assigned to the NAM speaker A by the assignment method shown in FIGS. In this state, when the NAM speaker A is silent for a certain time (Yes in step S50), the frequency / orientation allocating unit 32c maintains the allocated fundamental frequency and releases the allocated azimuth (step S60). Here, the release means that the allocation of NAM speaker A is canceled from the block allocated in step S10 without changing the “number of allocations” in the tables shown in FIGS.
On the other hand, if NAM speaker A is not mute for a certain period of time (No in step S50), the assigned fundamental frequency set in step S10 and until NAM speaker A becomes mute for a certain period of time (Yes in step S50) and The assigned orientation is maintained. Thereafter, the state in which the allocation direction is released is maintained until the NAM speaker A speaks again. When the NAM speaker A speaks again (in the case of Yes in step S70), the frequency / orientation allocating unit 32c is released from the block to be allocated to the NAM speaker A, that is, the allocated fundamental frequency maintained in step S60. It is determined whether or not a block having the assigned orientation can be assigned (step S80). If block allocation is possible (Yes in step S80), the frequency / orientation allocating unit 32c maintains the allocated fundamental frequency and resets the released allocation azimuth (step S90). If block allocation is impossible (No in step S80), the frequency / orientation allocation unit 32c changes the allocation fundamental frequency and allocation direction to the division frequency and division direction of the adjacent block (step S100). Thus, the NAM conversation is continued.

In the example shown in FIG. 11, the NAM speaker A speaks to the NAM conversation participant with the NAM. At this time, the frequency / orientation allocating unit 32c newly sets a basic frequency and an allocation direction to be assigned to the NAM speaker A by the assignment method shown in FIGS.
In this state, when the NAM speaker B who has high priority and is not assigned a block speaks in a NAM conversation (Yes in step S110), the frequency / orientation allocating unit 32c will assign to the NAM speaker B. It is determined whether or not the block to be overlapped with the block assigned to the NAM speaker A (step S120).
On the other hand, if it is determined in step S120 that the blocks to be assigned do not overlap (No in step S120), the frequency / orientation assigning unit 32c newly sets the assigned fundamental frequency and assigned orientation of the NAM speaker B. (Step S140). In the determination in step S120, when the allocation blocks overlap (in the case of Yes in step S120), the frequency / direction allocation unit 32c changes the allocation basic frequency and allocation direction of the NAM speaker A to the frequency and direction of the adjacent block. (Step S130). Thereafter, the process proceeds to step S140.
Thus, the NAM conversation is continued.

  Even when the NAM participant A approaches the NAM conversation participant P who is the listener, the new assignment in step S10 is performed. On the other hand, when the NAM speaker A moves away from the NAM conversation participant P or leaves the NAM conversation near the NAM conversation participant P, the NAM speaker A's assigned fundamental frequency and assignment are set. The direction is canceled. The withdrawal from the NAM conversation is executed when the NAM conversation management server 14 receives information indicating the withdrawal from the mobile terminal devices 12a to 12e.

As described above, the frequency / orientation allocating unit 32c determines the allocation direction and the allocation fundamental frequency for each NAM speaker from a plurality of blocks each including a plurality of division directions and a plurality of division frequencies as a set. One block is assigned to each NAM speaker so that no overlapping blocks are assigned to each other. For this reason, the setting of the allocation direction and the allocation fundamental frequency can be easily performed collectively.
Further, the frequency / orientation assigning unit 32c assigns one block to each NAM speaker using the current orientation of each NAM speaker and the voice fundamental frequency of each NAM speaker. Thus, the assigned orientation and assigned fundamental frequency can be set to the current orientation of each NAM speaker and the voice fundamental frequency of each NAM speaker.
When the block to be allocated has already been allocated to another NAM speaker, the frequency / azimuth allocation unit 32c is adjacent to the adjacent division frequency or adjacent division in which the division frequency or division direction of the block to be allocated is adjacent. Assign an adjacent block with orientation. For this reason, the allocation direction and the allocation fundamental frequency can be set to a direction and frequency close to the current orientation of each NAM speaker and the voice fundamental frequency of each NAM speaker.

  In the above allocation, priorities are determined in the order of the number of times of speech or the NAM speakers having a long time in a certain past time, and the allocation direction and the allocation fundamental frequency are determined for each NAM speaker according to this priority. For example, when a high priority NAM speaker moves and moves to a block assigned to another NAM speaker, the other NAM speaker already assigned loses the assigned block. Adjacent blocks are assigned. For this reason, it is possible to prevent duplication of blocks allocated to each NAM speaker in allocation.

As shown in FIG. 7, the direction and frequency allocation by the NAM conversation support method is performed when the NAM speaker is separated by a predetermined distance L ₁ or more as viewed from each NAM conversation participant A to E. Is called. When the NAM speaker is located within a preset distance L ₂ (see FIG. 7) as viewed from each NAM conversation participant A to E, the frequency / orientation assigning unit 32c assigns the current orientation of the NAM speaker. The direction is set as the direction, and the voice fundamental frequency of the NAM speaker is set as the assigned fundamental frequency. If the NAM speaker viewed from each NAM conversation participant A~E is located at a distance L ₂ within (see Fig. 7), since the NAM speaker is in the vicinity of each NAM conversation participant A~E, participate each NAM conversation The persons A to E can listen to the NAM speaker's NAM conversation from the direction that matches the current direction while watching the movement of the mouth and facial expression of the NAM speaker. Therefore, each NAM conversation participant A to E can listen to the NAM conversation while knowing who spoke.

When one NAM speaker approaches one NAM conversation participant, the frequency / orientation assigning unit 32c makes the assigned direction and the assigned fundamental frequency coincide with or approaches the current orientation and the voice fundamental frequency of the NAM speaker. Then, at least one of the allocation direction and the allocation fundamental frequency is corrected according to the distance between the NAM conversation participant and the NA speaker. Thus, as shown in FIG. 7, when the NAM speaker A is located within a distance L ₂ from the NAM conversation participant P, the assigned direction matches or approaches the current direction, and the assigned fundamental frequency is NAM speaker A. Therefore, the NAM conversation participant P can listen to the NAM conversation without feeling uncomfortable while watching the movement of the mouth and the facial expression of the approaching NAM speaker A.

  When one NAM speaker approaches one NAM conversation participant, the frequency / orientation assigning unit 32c smoothes at least one of the assigned orientation and the assigned fundamental frequency over a certain time, for example, 3 seconds as described above. Make corrections. For this reason, the NAM conversation participant can listen to the NAM conversation of the NAM speaker without discomfort.

  When the NAM speaker emits voiced voice and the microphone 16b picks up voiced voice, the voice mixing unit 32e cuts off the voiceless voice signal during the NAM conversation and converts the voiced voice signal into the communication unit 30. To the mobile terminal devices 12a to 12e. For this reason, it is possible to have a conversation using voiced voice together with the NAM conversation.

  The fundamental frequency calculation unit 32 obtains a speech fundamental frequency in advance by performing frequency analysis on the speech of each NAM speaker. For this reason, the person who can participate in the NAM conversation support system is not restricted.

  The said embodiment discloses the content shown below.

(Appendix 1)
A NAM conversation support system that supports NAM conversation by multiple people using NAM (Non Audible Murmur),
A measurement unit that acquires position information indicating the current position of each NAM conversation participant and information on a direction in which each NAM conversation participant is currently facing; a voice input / output device that each NAM conversation participant uses for NAM conversation; A mobile terminal device held by each NAM conversation participant, comprising: a communication unit that performs communication of the position information and the orientation information and communication of an audio signal during NAM conversation;
Based on the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker as seen from each NAM conversation participant specified using the position information and the orientation information received from the mobile terminal device, The NAM speaker's assigned direction and the NAM speaker's assigned fundamental frequency for each NAM conversation participant are determined, and the NAM silent voice signal received from the portable terminal device based on the assigned direction and the assigned fundamental frequency A NAM conversation support system, comprising: a conversation management device comprising: a signal processing unit to process; and a communication unit that transmits the processed silent voice signal to the portable terminal device.

(Appendix 2)
The signal processing unit, when determining the allocation direction and the allocation fundamental frequency for the NAM speaker, is separated from a plurality of blocks each having a set of a plurality of preset division directions and a plurality of preset division frequencies. The NAM conversation support system according to appendix 1, wherein one block is allocated to the NAM speaker so that no duplicate block is allocated to the NAM speaker.

(Appendix 3)
When the signal processing unit determines the allocation direction and the allocated fundamental frequency for the NAM speaker, the NAM speaker uses the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker as one block for the NAM. The NAM conversation support system according to appendix 2, which is assigned to a speaker.

(Appendix 4)
When the block to be allocated has already been allocated to another NAM speaker, the signal processing unit has an adjacent division frequency or adjacent division direction adjacent to the division frequency or division direction of the block to be allocated. The NAM conversation support system according to attachment 3, wherein one of the blocks is allocated.

(Appendix 5)
The signal processing unit preferentially assigns an adjacent block having a division frequency of the block to be assigned among adjacent blocks to an adjacent block having a division direction of the block to be assigned. The NAM conversation support system described in 1.

(Appendix 6)
In any one of appendices 1 to 5, the signal processing unit determines priority in the order of NAM speakers having a large number of times or time of speaking in a certain past time, and determines the allocation direction and the allocation fundamental frequency. The NAM conversation support system described.

(Appendix 7)
For the NAM speaker located within a predetermined distance range when viewed from each NAM conversation participant, the signal processing unit assigns the current orientation and the voice fundamental frequency of the NAM speaker to the assigned orientation and the assigned The NAM conversation support system according to any one of supplementary notes 1 to 6, which is set as a fundamental frequency.

(Appendix 8)
When the NAM speaker approaches one NAM conversation participant, the signal processing unit determines that the assigned orientation and the assigned fundamental frequency are the current orientation of the NAM speaker as viewed from the one NAM conversation participant. And at least one of the assigned orientation and the assigned fundamental frequency is modified according to the distance between the one NAM conversation participant and the NA speaker so as to match or approach the voice fundamental frequency, 8. The NAM conversation support system according to any one of 7 above.

(Appendix 9)
9. The NAM conversation support system according to appendix 8, wherein the signal processor corrects at least one of the allocation direction and the allocation fundamental frequency so as to change smoothly over a certain time.

(Appendix 10)
When the voice input / output device of the portable terminal device picks up voiced voice, the signal processing unit cuts off the voiceless voice signal during the NAM conversation and sends the voiced voice signal through the communication unit to the portable terminal. The NAM conversation support system according to any one of appendices 1 to 9, which is transmitted to a device.

(Appendix 11)
The NAM conversation support system according to any one of appendices 1 to 10, wherein the signal processing unit acquires the voice fundamental frequency in advance by performing frequency analysis on the voice of the NAM speaker.

(Appendix 12)
A NAM conversation support method performed by a conversation management device that supports NAM conversation by a plurality of people using NAM (Non Audible Murmur),
Receiving position information indicating the current position of each NAM conversation participant and information on the direction in which each NAM conversation participant is currently facing from a mobile terminal device held by each NAM conversation participant;
Each NAM conversation participant is based on the current direction of the NAM speaker and the voice fundamental frequency of the NAM speaker as viewed from each NAM conversation participant identified using the received position information and the orientation information. Defining the NAM speaker's allocation direction and the NAM speaker's allocation fundamental frequency for
A NAM conversation support method, comprising: processing a silent voice signal of NAM based on the assigned direction and the assigned fundamental frequency, and transmitting the processed silent voice signal to each NAM conversation participant.

(Appendix 13)
When the conversation management device determines the allocation direction and the allocation fundamental frequency for the NAM speaker, a plurality of blocks each of which is a set of a plurality of preset division directions and a plurality of preset division frequencies, 13. The NAM conversation support method according to appendix 12, wherein one block is allocated to the NAM speaker so that a block overlapping with the NAM speaker is not allocated.

(Appendix 14)
When the conversation management device determines the allocation direction and the allocated fundamental frequency for the NAM speaker, the NAM speaker uses one block using the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker. The NAM conversation support method according to appendix 13, which is assigned to

(Appendix 15)
When a block to be allocated by the conversation management apparatus is already allocated to another NAM speaker, an adjacent block having an adjacent divided frequency or adjacent divided direction adjacent to the divided frequency or divided direction of the block to be allocated 15. The NAM conversation support method according to appendix 14, wherein one of the above is assigned.

(Appendix 16)
The NAM conversation support according to appendix 15, wherein, among the adjacent blocks, an adjacent block having a division frequency of the block to be allocated is preferentially allocated to an adjacent block having a division direction of the block to be allocated. Method.

(Appendix 17)
17. The conversation management device according to any one of appendices 12 to 16, wherein the conversation management device sets priorities in order of NAM speakers having a large number of times of speaking or a long time in the past, and determines the allocation direction and the allocation fundamental frequency. NAM conversation support method.

(Appendix 18)
The conversation management device, for NAM speakers located within a predetermined distance range when viewed from each NAM conversation participant, assigns the current orientation and the voice fundamental frequency of the NAM speaker to the assigned orientation and the assigned 18. The NAM conversation support method according to any one of appendices 12 to 17, which is set as a fundamental frequency.

(Appendix 19)
When the NAM speaker approaches one NAM conversation participant, the conversation management device determines that the assigned orientation and the assigned fundamental frequency are the current orientation of the NAM speaker viewed from the one NAM conversation participant. And at least one of the assigned orientation and the assigned fundamental frequency is modified according to the distance between the one NAM conversation participant and the NA speaker so as to match or approach the voice fundamental frequency. 19. The NAM conversation support method according to any one of items 18.

(Appendix 20)
The NAM conversation support method according to supplementary note 19, wherein the conversation management device performs a smooth correction over a predetermined time when correcting at least one of the allocation direction and the allocation fundamental frequency.

(Appendix 21)
When the voice input / output device of the portable terminal device picks up voiced voice, the conversation management device cuts off the voiceless voice signal during the NAM conversation and transmits the voiced voice signal to the portable terminal device. The NAM conversation support method according to any one of appendices 12 to 20.

(Appendix 22)
The NAM conversation support method according to any one of appendices 12 to 21, wherein the conversation management device acquires the voice fundamental frequency in advance by performing frequency analysis of the voice of the NAM speaker.

  As described above, the NAM conversation support system and the NAM conversation support method of the present invention have been described in detail. However, the present invention is not limited to the above-described embodiment, and various improvements and modifications can be made without departing from the gist of the present invention. Of course it is also good.

DESCRIPTION OF SYMBOLS 10 NAM conversation support system 12,12a-12e Portable terminal device 14 NAM conversation management server 16 Headset 16a Headphone 16b Microphone 18 Measurement part 18a GPS position sensor 18b Direction sensor 20 Lombard detection part 22 Communication part 22a Voiced voice transmission part 22b NAM transmission Unit 22c Lombard transmission unit 22d position / azimuth transmission unit 22e audio signal reception unit 30 communication unit 30a voiced audio reception unit 30b NAM reception unit 30c Lombard reception unit 30d position / azimuth reception unit 30e audio signal transmission unit 32 signal processing unit 32a fundamental frequency Calculation unit 32b Database 32c Frequency / azimuth allocation unit 32d Whispering voice conversion unit 32e Audio mixing unit

Claims (9)

A NAM conversation support system that supports NAM conversation by multiple people using NAM (Non Audible Murmur),
A measurement unit that acquires position information indicating the current position of each NAM conversation participant and information on a direction in which each NAM conversation participant is currently facing; a voice input / output device that each NAM conversation participant uses for NAM conversation; A mobile terminal device held by each NAM conversation participant, comprising: a communication unit that performs communication of the position information and the orientation information and communication of an audio signal during NAM conversation;
Based on the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker as seen from each NAM conversation participant specified using the position information and the orientation information received from the mobile terminal device, The NAM speaker's assigned direction and the NAM speaker's assigned fundamental frequency for each NAM conversation participant are determined, and the NAM silent voice signal received from the portable terminal device based on the assigned direction and the assigned fundamental frequency A conversation management device comprising: a signal processing unit for processing; and a communication unit for transmitting the processed silent voice signal to the portable terminal device,
The signal processing unit, when determining the allocation direction and the allocation fundamental frequency for the NAM speaker, is separated from a plurality of blocks each having a set of a plurality of preset division directions and a plurality of preset division frequencies. A NAM conversation support system , wherein one block is allocated to the NAM speaker so that no duplicate block is allocated to the NAM speaker . When the signal processing unit determines the allocation direction and the allocated fundamental frequency for the NAM speaker, the NAM speaker uses the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker as one block for the NAM. assigned to speakers, NAM conversation support system according to claim 1. When the block to be allocated has already been allocated to another NAM speaker, the signal processing unit has an adjacent division frequency or adjacent division direction adjacent to the division frequency or division direction of the block to be allocated. The NAM conversation support system according to claim 2 , wherein one of the blocks is allocated. The signal processing unit preferentially allocates an adjacent block having a division frequency of the block to be allocated among adjacent blocks to an adjacent block having a division direction of the block to be allocated. 3. The NAM conversation support system according to 3. A NAM conversation support system that supports NAM conversation by multiple people using NAM (Non Audible Murmur),
A measurement unit that acquires position information indicating the current position of each NAM conversation participant and information on a direction in which each NAM conversation participant is currently facing; a voice input / output device that each NAM conversation participant uses for NAM conversation; A mobile terminal device held by each NAM conversation participant, comprising: a communication unit that performs communication of the position information and the orientation information and communication of an audio signal during NAM conversation;
Based on the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker as seen from each NAM conversation participant specified using the position information and the orientation information received from the mobile terminal device, The NAM speaker's assigned direction and the NAM speaker's assigned fundamental frequency for each NAM conversation participant are determined, and the NAM silent voice signal received from the portable terminal device based on the assigned direction and the assigned fundamental frequency A conversation management device comprising: a signal processing unit for processing; and a communication unit for transmitting the processed silent voice signal to the portable terminal device,
The signal processing unit may prioritize the order of the past number of times or time spoke constant time NAM speaker, determining the allocation orientation and the allocation fundamental frequency, NAM conversation support system, characterized in that. A NAM conversation support system that supports NAM conversation by multiple people using NAM (Non Audible Murmur),
A measurement unit that acquires position information indicating the current position of each NAM conversation participant and information on a direction in which each NAM conversation participant is currently facing; a voice input / output device that each NAM conversation participant uses for NAM conversation; A mobile terminal device held by each NAM conversation participant, comprising: a communication unit that performs communication of the position information and the orientation information and communication of an audio signal during NAM conversation;
Based on the current orientation of the NAM speaker and the voice fundamental frequency of the NAM speaker as seen from each NAM conversation participant specified using the position information and the orientation information received from the mobile terminal device, The NAM speaker's assigned direction and the NAM speaker's assigned fundamental frequency for each NAM conversation participant are determined, and the NAM silent voice signal received from the portable terminal device based on the assigned direction and the assigned fundamental frequency A conversation management device comprising: a signal processing unit for processing; and a communication unit for transmitting the processed silent voice signal to the portable terminal device,
For the NAM speaker located within a predetermined distance range when viewed from each NAM conversation participant, the signal processing unit assigns the current orientation and the voice fundamental frequency of the NAM speaker to the assigned orientation and the assigned A NAM conversation support system characterized by being set as a fundamental frequency. A NAM conversation support method performed by a conversation management device that supports NAM conversation by a plurality of people using NAM (Non Audible Murmur),
Receiving position information indicating the current position of each NAM conversation participant and information on the direction in which each NAM conversation participant is currently facing from a mobile terminal device held by each NAM conversation participant;
Each NAM conversation participant is based on the current direction of the NAM speaker and the voice fundamental frequency of the NAM speaker as viewed from each NAM conversation participant identified using the received position information and the orientation information. Defining the NAM speaker's allocation direction and the NAM speaker's allocation fundamental frequency for
The allocation orientation the assignment based on the fundamental frequency and processing the unvoiced speech signal NAM, transmits the unvoiced speech signal processed in each NAM conversation participant,
When the allocation direction and the allocation fundamental frequency are determined for the NAM speaker, the NAM speaker overlaps with another NAM speaker from a plurality of blocks each of which includes a plurality of preset division directions and a plurality of preset division frequencies. And assigning one block to the NAM speaker so that the assigned block is not assigned . A NAM conversation support method performed by a conversation management device that supports NAM conversation by a plurality of people using NAM (Non Audible Murmur),
Receiving position information indicating the current position of each NAM conversation participant and information on the direction in which each NAM conversation participant is currently facing from a mobile terminal device held by each NAM conversation participant;
Each NAM conversation participant is based on the current direction of the NAM speaker and the voice fundamental frequency of the NAM speaker as viewed from each NAM conversation participant identified using the received position information and the orientation information. Defining the NAM speaker's allocation direction and the NAM speaker's allocation fundamental frequency for
Processing a silent voice signal of a NAM based on the assigned direction and the assigned fundamental frequency, and sending the processed silent voice signal to each NAM conversation participant;
When determining the allocation direction of the NAM speaker and the allocation basic frequency of the NAM speaker, the priority is determined in the order of the number of NAM speakers or the number of NAM speakers having a long time, and the allocation direction and the allocation basic frequency. A NAM conversation support method, characterized by determining a frequency. A NAM conversation support method performed by a conversation management device that supports NAM conversation by a plurality of people using NAM (Non Audible Murmur),
Receiving position information indicating the current position of each NAM conversation participant and information on the direction in which each NAM conversation participant is currently facing from a mobile terminal device held by each NAM conversation participant;
Each NAM conversation participant is based on the current direction of the NAM speaker and the voice fundamental frequency of the NAM speaker as viewed from each NAM conversation participant identified using the received position information and the orientation information. Defining the NAM speaker's allocation direction and the NAM speaker's allocation fundamental frequency for
Processing a silent voice signal of a NAM based on the assigned direction and the assigned fundamental frequency, and sending the processed silent voice signal to each NAM conversation participant;
When determining the NAM speaker's allocation direction and the NAM speaker's allocated fundamental frequency, the NAM speaker's current status is determined for each NAM speaker located within a predetermined distance range as viewed from each NAM conversation participant. And setting the voice fundamental frequency as the assigned direction and the assigned fundamental frequency.