JP2016100677A - Presence transmission system and presence reproduction apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a presence transmission system capable of transmitting presence by reproducing a three-dimensional sound environment observed at a first location, for a person at a second location.SOLUTION: A presence transmission system 1000, on the transmission side, comprises: a microphone array group 100; a LRF group 200 for detecting a position of a person; and a sound source position determination apparatus 300 for estimating a direction in which sound arrives, identifying a position of a sound source by using integration with a detection result of position detection means, and separating sound arriving from the identified position of the sound source to output the separated sound. The presence transmission system, on the reception side, comprises: a face posture estimation unit 520 for estimating a face posture of a subject 2; and a sound synthesis unit 550 for synthesizing, depending on a position of a sound source and the face posture, a sound signal to be reproduced at each ear of the subject 2 from a signal of the separated sound by using a head transfer function from a second location's position corresponding to the position of the sound source at the first location to each ear of the subject.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a sound source localization technique and a sound space reconstruction technique, and more particularly to a technique for transmitting a sense of realism to a remote place using a sound source localization technique and a sound source separation technique.

  In recent years, in a robot remote control system, research for transmitting an operator's presence to a robot side has been widely performed (for example, see Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3). However, there are few studies that focus on transmitting a sense of realism to remote operators.

  Compared with face-to-face communication, when a person in a remote place communicates with a person via a robot, information shared with the other party is insufficient due to lack of spatial information or the like. For this reason, it is difficult for the operator to feel a sense of realism at the site where communication is performed.

  Virtual reality technology is a great help for the transmission of realism. Currently, virtual reality technology is used in many telemedicine, military, and communication applications, and the transmission of realism is one of these major purposes.

  On the other hand, there have already been many reports on sound source localization and sound source separation techniques in communication between robots and people. In voice communication between a person and a robot, the microphone attached to the robot is usually located at a distance (1 m or more). Therefore, the signal-to-noise ratio (SNR) is lower than when the distance between the microphone and the mouth is a few centimeters, for example, for telephone voices. For this reason, the voices of others nearby and the noise of the environment become disturbing sounds, making it difficult for the robot to recognize the target speech. Therefore, sound source localization and sound source separation are important for robot applications.

  Regarding the sound source localization, there are those described in Patent Document 1 or Patent Document 2 as conventional techniques assuming an actual environment. The technique described in Patent Document 1 or Patent Document 2 uses a known sound source localization method called the MUSIC method with high resolution.

  In the invention described in Patent Document 1 or Patent Document 2, a current correlation matrix is calculated based on a received signal vector obtained by Fourier-transforming a signal from the microphone array using a microphone array and a past correlation matrix. To do. The correlation matrix thus obtained is subjected to eigenvalue decomposition to obtain the maximum eigenvalue and a noise space that is an eigenvector corresponding to an eigenvalue other than the maximum eigenvalue. Furthermore, the direction of the sound source is estimated by the MUSIC method based on the phase difference of the output of each microphone, the noise space, and the maximum eigenvalue with one microphone as a reference in the microphone array.

  Furthermore, Patent Document 3 discloses a sound source localization and sound source separation system for the purpose of accurately separating human-generated speech and noise when humans and other noise sources coexist. Has been. Here, the sound source localization device includes an LRF (laser range finder) group that detects a person's position, each of a plurality of sound source signals obtained from the output of the microphone array group, and each microphone included in the microphone array. A sound source localization processing unit that calculates MUSIC power at predetermined time intervals for each of a plurality of directions based on the positional relationship and the output of the LRF group, and detects the peak as a sound source position at predetermined time intervals, and a microphone array The sound source separation processing unit that separates the sound signal from the sound source position detected by the sound source localization processing unit from the output signal of the sound, and the attribute of the separated sound signal is determined with high accuracy using the output of the human position measurement device And a sound source type identification processing unit.

Japanese Patent Application Laid-Open No. 2008-175733 JP 2011-220701 A Specification Japanese Patent Application Laid-Open No. 2012- 211768

Nishio, S., Ishiguro, H., Hagita, N. Can a Teleoperated Android Represent Personal Presence?-A Case Study with Children. Psychologia, 50 (4): 330-342. 2007. Ishi, CT, Liu, C., Ishiguro, H., Hagita, N. 2010.Head motion during dialogue speech and nod timing control in humanoid robots.In Proceedings of 5th ACM / IEEE International Conference on Hu-man-Robot Interaction ( HRI 2010). OSAKA, JAPAN. 293-300. Sumioka, H., Nishio, S., Minato, T., Yamazaki, R., Ishiguro, H. Minimal Human Design Approach for Sonzai-kan Media: Investigation of a Feeling of Human Presence. Cognitive Computation, 2014.

  However, most of these researches on virtual reality as described above focus on visual sense transmission. Although research on virtual reality related to the construction of sound environments is used in applications such as games, there are currently only a few.

A widely used method for reproducing a three-dimensional sound field is to reproduce binaural (both ears) recorded sound in stereo. This method has the advantage of being simple, but requires an accurate stereo microphone setting, and the dummy head does not move, so that the sound field cannot be reproduced dynamically. Furthermore, it is impossible to process each sound source.
Surround channel speakers have been developed to reproduce spatial sound fields, and there are many studies on sound field reproduction using DirAC (Directional Audio Coding).

  However, there are two problems with the surround speaker system. First, if the environment in which the sound field is recorded and the environment in which it is played back are different, environmental factors such as the size and shape of the room affect the sound transmission, and these effects are corrected accurately. That is difficult. Second, in the surround speaker system, the “sweet spot” position is limited near the center of the system. That is, the location of the listener is limited.

  Despite these circumstances, the creation of a rich sound environment on the reproduction side is intended to convey the presence of the remote location to the operator and the presence of the site even in social media such as remote control robots. It is considered an important factor.

  The present invention has been made to solve such a problem, and the object thereof is to provide a person in a second place with a three-dimensional sound environment observed in the first place. It is to provide a sense of presence transmission system and a sense of presence reproduction device capable of transmitting a sense of reality by reproducing the actual feeling.

  According to one aspect of the present invention, there is provided a sense of presence transmission system for transmitting and reproducing the sound environment of the first place to the second place, comprising a sound source localization device installed at the first place. The sound source localization device estimates the direction of arrival of sound according to the output from the microphone array unit and position detection means for detecting the position of the object in the first place, and integrates it with the detection result of the position detection means. And a sound source localization unit for specifying and outputting the position of the sound source, and a sound source separation unit for separating and outputting the sound from the specified position of the sound source, and being synthesized at the second location The speech synthesizer further includes a face posture detecting means for detecting the face posture of the subject in the second location, and a sound that is attached to the subject and that corresponds to the sound environment with respect to both ears of the subject. Sound reproduction means for reproduction and sound source localization means The position of the position of the sound source is received, and the head transmission from the position corresponding to the position of the position of the sound source at the first place to each ear of the subject in the second place according to the detected face posture. Sound space reconstruction means for synthesizing a sound signal to be reproduced to each ear by the sound reproduction means from the separated sound signal from the sound source separation means using a function.

  Preferably, the sound reproduction means is headphones, and the face posture detection means includes a gyro and a compass attached to the headphones.

  Preferably, the sound reproduction means is a headphone, and the face posture detection means estimates the face posture of the subject from the captured subject image.

  Preferably, the microphone array unit includes a plurality of microphone arrays, and the sound source localization means responds to the fact that the sound arrival direction based on each of the plurality of microphone arrays and the position of the sound source detected by the position detection means intersect. To specify the position of the sound source.

  Preferably, the apparatus further includes a database that stores coefficients of a plurality of head-related transfer functions corresponding to directions from the sound source to each ear of the subject, and the sound space reconstructing means is the sound source at the first location at the second location. A head-related transfer function from the position corresponding to the position of the head to each ear of the subject is selected from the database, and a sound signal for reproduction to each ear is synthesized.

  Preferably, the sound space reconstruction device is configured such that the subject specifies the position or face posture of the subject in the second place, and the volume of the signal of the separated sound from the sound source separation unit according to the instruction from the instruction unit And volume control means for individually controlling the sound volume.

  According to another aspect of the present invention, the realistic reproduction device for reproducing the sound environment of the first place at the second location based on the information from the transmission device that transmits the information about the sound environment of the first location. The transmitting device transmits position information indicating the position of the sound source in the first place and a separated sound signal obtained by separating the sound from the position of the sound source specified by the position information. A face posture detection means for detecting the face posture of the subject in the place, a sound reproduction means for reproducing sound corresponding to the sound environment for both ears of the subject, and a position of the sound source position The information is received and separated according to the detected face posture using a head-related transfer function from the position corresponding to the position of the sound source at the first place to each ear of the subject at the second place. Sound signal for reproducing to each ear from sound signal by sound reproduction means And a sound space reconstruction means for combining.

  Preferably, the sound reproduction means is headphones, and the face posture detection means includes a gyro and a compass attached to the headphones.

  Preferably, the sound reproduction means is a headphone, and the face posture detection means estimates the face posture of the subject from the captured subject image.

  Preferably, the apparatus further includes a database that stores coefficients of a plurality of head-related transfer functions corresponding to directions from the sound source to each ear of the subject, and the sound space reconstructing means is the sound source at the first location at the second location. A head-related transfer function from the position corresponding to the position of the head to each ear of the subject is selected from the database, and a sound signal for reproduction to each ear is synthesized.

  Preferably, the subject individually controls the volume of the separated sound signal from the sound source separation means in accordance with the instruction means for specifying the position or face posture of the subject in the second place and the instruction from the instruction means. Volume control means is further provided.

  According to the present invention, it is possible to convey a sense of realism by reproducing the three-dimensional sound environment observed at the first place with respect to the person at the second place.

  Further, according to the present invention, it is possible to transmit a sense of reality of the environment where the robot exists to an operator who operates the remote control type robot.

It is a block diagram for demonstrating the structure of the presence transmission system 1000 of this Embodiment. 3 is a functional block diagram for explaining a configuration of a sound source localization apparatus 300. FIG. It is a functional block diagram for demonstrating the speech synthesizer 500 of the receiving side. 4 is a block diagram for explaining a hardware configuration of a sound source localization apparatus 300. FIG. It is a figure for demonstrating an experimental system. It is a figure which shows an experimental result. It is a figure which shows the example of a screen display of an interface. It is a figure which shows the experimental result of a user interface.

  Hereinafter, the configuration of the presence transmission system according to the embodiment of the present invention will be described with reference to the drawings. In the following embodiments, components and processing steps given the same reference numerals are the same or equivalent, and the description thereof will not be repeated unless necessary.

  In the following description, a so-called microphone, more specifically an electret condenser microphone, will be described as an example of the sound sensor. However, any other sound sensor may be used as long as it can detect sound as an electric signal. Also good.

  As will be described below, the realistic sensation transmission system according to the present embodiment is configured such that a sound environment (three-dimensional sound field) composed of a plurality of sound sources distributed around a robot in a remote place is represented by an operator (operator). ) Reproduce and process on the side to convey the sense of reality of the sound.

  The headphone is taken as an example for reproducing the sound environment on the operator side.

  FIG. 1 is a block diagram for explaining the configuration of the realistic sensation transmission system 1000 according to the present embodiment.

  In FIG. 1, it is assumed that the transmission-side coordinate system of the presence is (x, y, z), and the reception-side coordinate system is (x ′, y ′, z ′).

  In the presence transmission system 1000, the transmission side of the presence is a plurality of microphone arrays 10.1-10. A microphone array group 100 including M, and a plurality of laser range finders (LRF) 20.1 to 20. LRF group 200 including L, and sound source localization apparatus 300 that performs sound source localization and sound source separation existing in the environment on the transmission side based on the outputs of microphone array group 100 and LRF group 200.

  In the sound source localization apparatus 300, the human position detection / tracking unit 310 uses the output of the LRF group 200 to detect information (referred to as human position information) indicating a position where a person exists, and responds to the movement of the person. Thus, the human position is tracked even during the non-voicing period. The sound source localization unit 320 receives the output of the microphone array group 52 and the human position information output from the human position detection / tracking unit 310, performs sound source localization based on the audio signal output from the microphone array group 52, and performs sound source separation. The unit 330 collects sound from each sound source that is separated from the sound source, and transmits the separated sound to the receiving side. Information on the direction and position of the sound source (referred to as direction / position information) from the sound source localization unit is also transmitted to the receiving side.

  In processing on the reception side (operator side: presence reproduction device) of the presence transmission system 1000, the speech synthesizer 500 receives the separated sound from the sound source separation unit 330 and normalizes the volume. Then, based on the information from the sensor 600 on the headphones worn by the operator 2, the face posture estimation unit 520 that estimates the face direction of the operator 2 and the received direction / position information of the face of the operator 2 estimated. Depending on the direction, the head transfer function (HTRF: Head Relative Transfer Function) corresponding to the left and right channels is selected from the database 530 from the position of the sound source and the direction of the face, convolution is performed on the separated sound, and stereo headphones 610 includes a sound space reconstruction unit 540 that reconstructs and synthesizes the sound to be played back to the operator 2 at 610.

  As the sensor 600 for tracking the head rotation of the operator 2, a gyro sensor and a compass attached to the upper part of the headphone 610 can be used.

  Further, the volume control unit 310 may be configured such that the operator 2 can independently adjust the volume of each separated sound source through a user interface displayed on the display unit 650.

  Note that the processing on the transmission side and the reception side may be configured to provide a synchronization time server for controlling the synchronization of each unit constituting the system, and to process each process in synchronization.

  FIG. 2 is a functional block diagram for explaining the configuration of the sound source localization apparatus 300.

  Referring to FIG. 2, sound source localization section 320 includes microphone arrays 10.1-10. 3D spatial DOA evaluation units 3202.1 to 3202,... That estimate the direction of arrival (DOA: Direction Of Arrival) of the sound by signals from M respectively. M and a 3D spatial map storage unit 3204 for storing a 3D spatial map. The spatial information integration unit 3206 includes a positional relationship between the environment represented by the 3D spatial map and the microphone array, DOA of each sound source, and By integrating the information from the human position detection / tracking unit 310, the human position information in three dimensions is acquired. This person position information is always tracked by the person position detection and tracking unit 310 constituting the human tracking system even during non-speech.

  In the sound source separation unit 330, the sound source separation processing units 3302.1 to 3302. n separates each person's voice based on the estimated person position information, and sends it to the receiving side (operator side) system together with the position information from the spatial information integration unit 3206.

Hereinafter, the operation of each unit will be described in more detail.
(3D sound source localization)
Regarding sound source localization, first, the three-dimensional space DOA evaluation unit 3202.1 to 3202. M represents each microphone array 10.1-10. DOA estimation is performed for each of M. The spatial information integration unit 3206 estimates the position of the sound source in the three-dimensional space by integrating the DOA information from the plurality of arrays and the human position information from the human position detection tracking unit 310.
The DOA estimation of sound in a real environment has been widely studied, and the MUSIC method is one of the most effective methods that can localize a plurality of sources with high resolution. It is disclosed. Assuming that the number of sound sources is fixed, the peak of the MUSIC spectrum exceeding the threshold is recognized as a sound source. Here, for example, even when the MUSIC method is implemented so as to have a resolution of once every 100 ms, the direction of the sound source can be searched in real time by a single core CPU having an operation clock frequency of 2 GHz.
Furthermore, the most important sound source for a communication robot remote control system is human voice. Therefore, the sound source localization apparatus 300 uses a human tracking system configured by a plurality of two-dimensional LRFs in order to extract a human voice without omission. If the DOA estimation output from the plurality of microphone arrays and the LRF tracking result intersect at the same position, the spatial information integration unit 3206 determines that there is a high possibility that there is a sound source there.

Here, when the two-dimensional LRF is used as in the sound source localization apparatus 300, the human position information is limited to two dimensions. Here, the sound source is specified by limiting the position of the detected sound source within the range of the height of the mouth (for example, z = 1 to 1.6 m). In silent sections and sections where sound source direction estimation is insufficient, sound source separation is performed using the last estimated mouth height and the latest two-dimensional position information.
(Sound source separation)
The sound source separation unit 330 separates a plurality of selected persons (number of persons: n persons) in parallel.

Sound source separation processing units 3302.1 to 3302. In n, the human voice in the target direction is separated using a delay-sum beamformer with a small amount of calculation and a robustness. The frame length is 20 ms and the shift length is 10 ms.
Here, the delay sum beamformer is disclosed in the following document, for example.

Reference 1: International Publication WO 2004/034734 (Republished 2004-034734)
The basic principle of beam forming will be briefly described by taking the case of 2 microphones as an example.

Consider a situation in which two omnidirectional microphones having exactly the same characteristics are arranged at an interval d and a plane wave arrives from a direction θ. This plane wave is received by each microphone as a signal having a different propagation delay time by the path difference dsinθ. In a beam former that is an apparatus that performs beam forming, one microphone signal is delayed by δ = dsin θ ₀ / c (c is the speed of sound) so as to compensate for a propagation delay related to a signal arriving from a certain direction θ _0. The output signal is added to or subtracted from the other microphone signal.

At the input of the adder, the phases of the signals arriving from the direction θ ₀ match. Accordingly, the signal coming from the direction θ ₀ is emphasized at the output of the adder. On the other hand, signals coming from directions other than θ ₀ are not emphasized as much as signals coming from θ ₀ because their phases do not match each other. As a result, the beamformer using the adder output forms a directivity having a beam (Beam: a particularly sensitive direction) at θ ₀ . In contrast, the subtractor completely cancels the signal coming from direction θ ₀ . Therefore, the beamformer using the subtracter output forms a directivity having null (Null: a direction with particularly low sensitivity) at θ ₀ . A beamformer that performs only delay and addition in this way is called a “delay sum beamformer”.

  Here, more generally, assuming that a directional sound source S and an omnidirectional noise source N are present in space, the output of the delayed sum beamformer has the following form:

Y is the output of the beamformer corresponding to the frequency f, Sdir indicates the signal direction, and w _Sdir indicates the beamformer response in the Sdir direction. The second item in the equation represents noise mixed in separated speech. In order to reduce this noise component, the following weights are applied to each frequency.

_YPF is the beamformer output after weighting.
FIG. 3 is a functional block diagram for explaining the speech synthesizer 500 on the receiving side.

  The volume control unit 510 receives the separated sounds from the sound source separation unit 330 and normalizes the volume, respectively. n.

  The volume control unit 510 performs normalization on each separated sound as follows in order to correct a difference due to the distance between each sound source and the array.

Of these, N is the number of sound sources, and dist _n is the distance between the nth sound source and the array. g _i is a normalization factor applied to the i-th sound source, and Y i indicates the separation result of the i-th sound source.

  The face posture estimation unit 520 estimates the face direction of the operator 2 based on information from the sensor 600 on the headphones worn by the operator 2.

  However, for example, the method of estimating the face direction of the operator 2 is not limited to such a configuration. For example, an image of the operator 2 is captured and the head posture of the operator 2 is estimated from the captured data. It is good to do. The estimation of the head posture based on such a captured image is not particularly limited, but is disclosed in the following document, for example.

Document 2: JP 2014-93006 A In the sound space reconstruction unit 540, the space reconstruction unit 550 determines the coordinate system according to the direction / position information received from the transmission side and the estimated face orientation of the operator 2. The position of the sound source at (x ′, y ′, z ′) is reconstructed, and an accurate head transfer function (HTRF: Head Relative Transfer Function) corresponding to the left and right channels is calculated from the estimated face orientation in the database 530. Select from.

  Here, the head-related transfer function HTRF is an impulse response obtained by measuring an impulse signal emitted from an arbitrarily arranged sound source at the listener's ear canal entrance, and is disclosed in the following documents, for example.

Document 3: JP 2010-118978 A In the sound space reconstruction unit 540, the HTRF processing units 5502.1 to 5502. n performs a convolution operation with the selected head-related transfer function on the separated and volume-controlled speech, and the left ear speech synthesis unit 5504.1 and the right ear speech synthesis unit 5504.2 are connected to the stereo headphone 610. The left ear sound and the right ear sound to be reproduced to the operator 2 by the left and right speakers are respectively synthesized.

  In reproduction of a 3D sound field using headphones, it is used that a person ordinarily performs sound source localization based on a difference in sound waves reaching both ears. By reproducing this difference with the headphones 610, it is possible to synthesize a 3D sound field with stereo headphones.

  The head-related transfer function HTRF is a function that expresses the difference in the point in time when sound waves emitted from a sound source in space reach the human ears, and is often used to reproduce a 3D sound field. However, when reproducing the sound source existing in the space using headphones, there is a problem that the virtual sound source moves with the movement of the listener's head and body. Considering human daily experience, the position of the external sound source is not related to the movement of the listener's body and is fixed. Reproduction of 3D sound field using headphones is different from this experience, so it works negatively in the transmission of realism and causes an unnatural impression. Further, when the head-related transfer function is used, there is a problem that a wrong judgment before and after occurs. This sounds like a sound source in front is behind, or vice versa. In everyday life, the head is turned consciously and unconsciously to localize the sound source, and the effect is used to assist in localization.

  Taking these into consideration, the presence-sensation transmission system 1000 synthesizes stereo sound using HTRF that matches the direction of the head by tracking the head rotation of the operator 2. The continuous sound source position information necessary for selecting an accurate HTRF is obtained from DOA estimation results of a plurality of microphone arrays and a human position estimation system.

  That is, in order to make one sound heard from a specific direction, the sound is filtered and stereo- lated by the HTRF corresponding to that direction. A database of coefficients representing HTRF is not particularly limited, but, for example, a publicly available KERF (Knowles Elec-tronics Manikin for Acoustic Research) dummy head HTRF database can be used. KEMAR is a dummy head made using a general head size for HTRF research. The database shows the response of the left and right ears of the dummy head to impulse signals from the space, from an elevation angle of -40 degrees to 90 degrees. The impulse response in a total of 710 directions is included. Each impulse response has a length of 512 samples and a sampling frequency of 44.1 kHz. It is also possible to synthesize HTRF corresponding to the shape of the subject's head and use it as a database.

  In order to dynamically synthesize a sound field using HTRF, real-time detection of the head orientation is necessary. Therefore, as described above, a gyro sensor and a compass are attached to the upper part of the headphone to track the head rotation. It can be configured. At this time, the angle information is sent to the system either serially or via Bluetooth. The direction used to synthesize the sound field is the sound source direction minus the head angle, and the impulse responses of the left and right channels corresponding to this direction are selected from the database, and the result of the separation and convolution calculation is the operator's ears. To be played.

  FIG. 4 is a block diagram for explaining a hardware configuration of the sound source localization apparatus 300.

  Note that the speech synthesizer 500 basically has the same configuration. That is, the function of each functional block shown in FIG. 2 or FIG. 3 is realized by software that operates on hardware as described below.

  As shown in FIG. 4, the sound source localization device 300 includes a drive device 52 that can read data recorded on the external recording medium 64, and a central processing unit (CPU) 56 connected to a bus 66. ROM (Read Only Memory) 58, RAM (Random Access Memory) 60, nonvolatile storage device 54, microphone array 10.1-10. Audio data from M and laser range finder 20.1-20. A data input interface (hereinafter referred to as data input I / F) 68 for fetching distance measurement data from L is included.

  As the external recording medium 64, for example, an optical disk such as a CD-ROM or a DVD-ROM or a memory card can be used. However, the target recording medium is not limited to this as long as the device that realizes the function of the recording medium drive 52 is a device that can read data stored in a nonvolatile recording medium such as an optical disk or a flash memory. In addition, a device that realizes the function of the nonvolatile storage device 54 may be a magnetic storage device such as a hard disk or a flash memory as long as it can store data in a nonvolatile manner and can be accessed randomly. A solid state drive (SSD) that uses a nonvolatile semiconductor memory such as a storage device can also be used.

  The main part of such a sound source localization apparatus 300 is realized by computer hardware and software executed by the CPU 56. In general, such software is recorded at the time of manufacture of the sound source localization device 300 by a mask ROM, a programmable ROM, or the like, and may be read into the RAM 60 at the time of execution, or may be read from the recording medium 64 by the drive device 52. A configuration may be adopted in which the data is read and temporarily stored in the nonvolatile storage device 54 and then read out to the RAM 60 at the time of execution. Alternatively, when the device is connected to a network, the server is temporarily copied from the server on the network to the nonvolatile storage device 54, read from the nonvolatile storage device 54 to the RAM 60, and executed by the CPU 56. There may be.

  The computer hardware itself and its operating principle shown in FIG. 4 are general. Accordingly, one of the most essential parts of the present invention is software stored in a recording medium such as the nonvolatile storage device 54.

In the case of the speech synthesizer 500, the database 530 can also be stored in the nonvolatile storage device 54.
(System evaluation experiment)
In the following, a subject experiment conducted for evaluating the realistic feeling transmission system 1000 will be described.

  FIG. 5 is a diagram for explaining such an experimental system.

  FIG. 5C shows the environment on the operator side, and FIG. 5B shows the environment on the robot side.

The subject (operator) shown in FIG. 5 (c) has a conversation with a person on the robot side (person in FIG. 5 (b)) via the robot, and the conversation partner without visual information on the robot side. It is required to estimate the direction of the road.
As a comparison object, a stereo microphone located at the ear of the robot shown in FIG. In this experiment, a minimalist humanoid robot Telenoid-R3 was used. Since this robot can be equipped with microphones at both ear positions and has three degrees of freedom in the neck, the head movement of the subject in FIG. 5C can be linearly mapped.
The conditions for comparison are described below. Under this condition, the sound taken from the two microphones in the robot's ear is played as it is on the left and right channels of the operator's stereo headphones. The tracked operator's neck movement is linearly mapped to the robot.
The three-dimensional sound source position estimation on the robot side was performed by three microphone arrays indicated by white arrows in FIG. Two microphone arrays with a 15 cm diameter 8-channel microphone arranged in a circle are installed on the ceiling, and a microphone array with a 30 cm diameter 16-channel microphone arranged on a hemisphere is installed on the desktop. is there.
A total of 20 subjects participated in this experiment. All are university students who are not involved in robotics or acoustic research. The subject was instructed as an operator to talk to a speaker (research assistant) on the robot side in a separate room via the robot and determine the direction of the other person. The experiment assistant chooses a direction at random and advances the conversation from that direction. The subject informs the collaborator when the direction is determined, and the collaborator moves in the next direction. This procedure was repeated 4 times. Direction determination is limited to eight directions, and the subject answers in which direction.
At the end of the experiment, we took a subjective evaluation questionnaire on the presence and ease of listening for the two conditions. In a seven-step evaluation from 1 to 7, 1 indicates “low sense of reality / difficult to hear” and 7 indicates “high sense of reality / easy to hear”.
FIG. 6 is a diagram showing experimental results.

  FIG. 6A shows the average value of the direction localization accuracy under the condition in the realistic sensation transmission system 1000 and the comparison condition, and the standard deviation thereof.

As a result of t-test, a significant difference was found in the accuracy difference between the two (t = 0.59, p <0.001).
As shown in FIGS. 6B and 6C, in the subjective evaluation questionnaire, similar results were obtained in the evaluation of the presence and ease of hearing. In both the sense of presence and the ease of hearing, the evaluation with the condition in the presence transmission system 1000 is significantly higher than the comparison condition (t = 6.68, p <0.001 and t = 4.86, p <0.001).
One possible reason for the significant difference between the two conditions in the evaluation of the presence is that the movable range of the robot's neck and the human's neck are different.
(Adjustment of sound source volume in virtual sound field)
In the realistic sensation transmission system 1000, the stereo sound reflecting the position information is synthesized and added to all the selected sound sources, and the output representing the virtual sound field is reproduced. However, this cannot predict the volume of each selected sound source. If the operator can control the volume of each sound source independently, the most comfortable sound environment for him can be created. With this in mind, an interface may be provided so that the operator can change the sound source and his / her position in the virtual space.
Below, the user interface of two different operation patterns for controlling a virtual sound field is demonstrated.

FIG. 7 is a diagram showing a screen display example of such an interface.
In the first interface shown in FIG. 7A, the operator displays a white circle on the screen (this represents the position of the operator on the virtual space (coordinate system (x ′, y ′, z ′)). Adjust the volume of each sound source by dragging and dropping it to the desired location. A black circle indicates the position of the assistant for the experiment.

By moving your own virtual position to the desired location, the distance and angle with each sound source are recalculated, and the volume of the sound source is changed according to the distance (when approaching a specific sound source, the volume of that sound source growing). This interface is expressed as “drag-and-drop”. In a conversation scene in a real environment, the physical distance between conversation participants is influenced by the environment and social relationships with the other party. “Drag-and-drop” is a virtual sound field control method that focuses on this viewpoint.
In the second interface shown in FIG. 7B, the volume of each sound source is adjusted according to the orientation of the operator's face. Both hands are released to control the volume of the sound source using the face direction of the operator. The sound source in front of the operator's face is emphasized and the sound source in the back is attenuated. The factor that adjusts the volume is proportional to the angle. This interface is denoted as “face dir”. The direction of the face and the direction of the line of sight not only indicate the attention of the person at the present time, but also indicate the next target and its goal. “Face dir” is a virtual sound field control method that focuses on this point of view.
In FIG. 7B, the direction of the face of the subject (operator) is indicated by an arrow attached to the white circle.
(Evaluation of the proposed user interface)
A subject experiment was conducted to evaluate the user interface of FIG. For comparison, an interface using a conventional monaural microphone was used.
The experimental subjects described in FIG. 5 also participated in this experiment (16 university students. The first 4 out of 220 in the previous section were excluded because they were not compared with the conventional method).

The experimental design used within-subject comparison. The subject uses the proposed interface and the conventional interface to converse with two interlocutors (research assistants) in the robot environment. There are no restrictions on conversation topics. Separate conversation sessions for each interface used. The length of the session was 3 minutes, and after each session, a seven-stage subjective evaluation questionnaire from 1 to 7 was taken in the same way as in the previous experiment regarding “ease of use”, “realism”, and “easy to hear” of the interface.
FIG. 8 is a diagram illustrating a result of the user interface experiment.

FIG. 8 shows the average value and standard deviation of subjective evaluation for each interface. Analysis of variance (ANOVA, with-in participants, Bonferroni's posttest) was performed on the experimental results.
There was a significant difference in the average value of subjective evaluation between “ease of use” shown in FIG. 8A and “realism” in FIG. 8B (F (2,13) = 16.03, p <0.001). and F (2,13) = 6.74, p = 0.009).

As a result of multiple comparison (Bonferroni method), “drag-and-drop” and “face dir” are easier to use than the conventional method (“drag-and-drop” vs. “conventional”: p = 0.001; “face
dir ”vs.“ conventional ”: p = 0.001) and high presence (“ drag-and-drop ”vs.“ conventional ”: p = 0.006;“ face dir ”vs.“ conventional ”: p = 0.04) It was evaluated.

However, there was no significant difference in “easy to hear” (F (2,13) = 3.67, p = 0.052).
The above results show the effectiveness of the proposed interface.

  In the above description, in the realistic sensation transmission system 1000, it has been described that the position of the sound source on the transmission side is specified by a person. However, the present invention is not limited to such a case, and the presence sensation transmission is performed. The system 1000 can be used to reproduce a sound environment including a sound source that generates sound while moving on the receiving side.

  As described above, according to the realistic sensation transmission system 1000, the realistic sensation is transmitted by reproducing the three-dimensional sound environment observed in a predetermined place for a person in a different place. It is possible.

  Further, according to the presence transmission system 1000, it is possible to transmit the presence of the environment in which the robot exists to an operator who operates the remote operation type robot.

  Further, according to the presence sense transmission system 1000, it is possible to experience while changing the sense of presence according to the will of the subject by controlling how the sound is heard at a remote place by the user's operation. .

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  2 Subjects, 10.1-10. M microphone array, 20.1-20. L LRF, 100 microphone array group, 200 LRF group, 300 sound source localization device, 310 person position detection tracking unit, 320 sound source localization unit, 330 sound source separation unit, 500 speech synthesizer, 510 volume control unit, 520 face posture estimation unit, 530 database, 540 spatial speech reconstruction unit, 550 speech synthesis unit, 600 sensor, 610 headphones, 650 display unit.

Claims (11)

A realistic feeling transmission system for transmitting and reproducing the sound environment of the first place to the second place,
A sound source localization device installed in the first location, the sound source localization device,
Position detecting means for detecting the position of the object in the first place;
According to the output from the microphone array unit, the direction of sound arrival is estimated, integrated with the detection result of the position detection means, and the sound source localization means for specifying and outputting the position of the sound source;
Sound source separation means for separating and outputting the sound from the position of the identified sound source,
A speech synthesizer installed in the second location;
Face posture detection means for detecting the face posture of the subject in the second location;
Sound reproduction means for reproducing the sound corresponding to the sound environment for both ears of the subject, worn by the subject;
The position of the position of the sound source is received from the sound source localization means, and the position of the sound source in the second place is determined from the position corresponding to the position of the sound source of the first place according to the detected face posture. Sound space reconstruction means for synthesizing a sound signal for reproduction to each ear by the sound reproduction means from a separated sound signal from the sound source separation means using a head-related transfer function to each ear of the subject. , Presence transmission system. The sound reproduction means is a headphone,
The presence sense transmission system according to claim 1, wherein the face posture detection means includes a gyro and a compass attached to the headphones. The sound reproduction means is a headphone,
The presence sense transmission system according to claim 1, wherein the face posture detection unit estimates the face posture of the subject from the captured image of the subject. The microphone array unit includes a plurality of microphone arrays,
The sound source localization means identifies the position of the sound source in accordance with a crossing of a sound arrival direction based on each of a plurality of microphone arrays and a sound source position detected by the position detection means. The realistic sensation transmission system according to any one of to 3. A database further storing coefficients of a plurality of head related transfer functions according to directions from the sound source to each ear of the subject;
The sound space reconstruction means includes:
In the second place, a head-related transfer function to the ear of the subject from a position corresponding to the position of the sound source in the first place is selected from the database and reproduced to each ear. The realistic sensation transmission system according to claim 1, wherein the sound signal is synthesized. The speech synthesizer
Instructing means for the subject to specify his or her position or face posture in the second location;
The sound volume control means for individually controlling the sound volume of the signal of the separated sound from the sound source separation means in accordance with an instruction from the instruction means, according to any one of claims 1 to 5. Realistic transmission system. A realistic reproduction device for reproducing the sound environment of the first location at a second location based on information from the transmission device that transmits information about the sound environment of the first location, from the transmission device Is transmitted position information indicating the position of the sound source in the first location, and a signal of the separated sound obtained by separating the sound from the position of the sound source specified by the position information,
Face posture detection means for detecting the face posture of the subject in the second location;
Sound reproduction means for reproducing the sound corresponding to the sound environment for both ears of the subject, worn by the subject;
The position information of the sound source position is received, and from the position corresponding to the position of the sound source position of the first place to each ear of the subject in the second place according to the detected face posture. And a sound space reconstruction unit that synthesizes a sound signal for reproduction to each ear by the sound reproduction unit from the separated sound signal using the head-related transfer function. The sound reproduction means is a headphone,
The realistic sensation reproduction apparatus according to claim 7, wherein the face posture detection means includes a gyro and a compass attached to the headphones. The sound reproduction means is a headphone,
The realistic sensation reproduction apparatus according to claim 7, wherein the face posture detection unit estimates the face posture of the subject from the captured image of the subject. A database further storing coefficients of a plurality of head related transfer functions according to directions from the sound source to each ear of the subject;
The sound space reconstruction means includes:
In the second place, a head-related transfer function to the ear of the subject from a position corresponding to the position of the sound source in the first place is selected from the database and reproduced to each ear. The realistic sensation reproducing apparatus according to claim 7, wherein the sound signal is synthesized. Instructing means for the subject to specify his or her position or face posture in the second location;
The sound volume control means for individually controlling the sound volume of the separated sound signal from the sound source separation means in accordance with an instruction from the instruction means, according to any one of claims 7 to 10. Realistic reproduction device.