JP5754595B2 - Trans oral system - Google Patents

Description

  The present invention relates to a transaural system that can cope with changes in the posture of a listener during sound reproduction in communication / broadcasting.

  The trans-oral system is a system that realizes a method of generating a signal equivalent to a signal reaching the listener's ears in the original sound field in the listener's ears by a speaker in the reproduction sound field. The trans-oral system is expected in the field of virtual reality, for example, because it is possible to listen to the auditory information of the virtual world and the surrounding sound in the real world in an overlapping manner.

  Humans have sound image localization ability that can perceive the direction and distance of sound. This ability is empirically realized by estimating the transfer function (head-related transfer function) from the sound source to the left and right ears from the left and right sounds. When the head-related transfer function is convolved with a monaural sound source and input to both ears, a sound image can be given from the position of the sound indicated by the head-related transfer function. When a trans-oral system is realized, since the sound from one speaker reaches both ears, it is necessary to remove the crosstalk component in consideration of the transfer function from the speaker to both ears.

  As a trans-oral system corresponding to fluctuations in the position of the head, the one disclosed in Non-Patent Document 1 is known. FIG. 6 shows a system configuration of the trans-oral system 900 of Non-Patent Document 1. The trans-oral system 900 includes a pair of speakers 91 and 92, a sensor 93, a head position detection unit 94, a binaural processing unit 95, a crosstalk processing unit 96, a D / A converter 97, and an amplifier 98. It has.

  The sensor 93 is attached to a listener's head positioned in front of the pair of speakers 91 and 92 and detects a change in the position of the listener's head. For example, a magnetic sensor is used as the sensor 93, and position change information with three degrees of freedom is output.

  The head position detection unit 94 detects head position information changed from the front of the pair of speakers 91 and 92 from the position change information output from the sensor 93. The binaural processing unit 95 receives the audio signal and the head position information as input and synthesizes a binaural signal from the audio signal (digital signal). A binaural signal is a sound signal that can be heard three-dimensionally when listening through headphones. The head signal transfer function from the right speaker to the right ear of the listener and the left speaker to the listener's left ear Are the left and right channel signals obtained by convolving and synthesizing the head related transfer functions. The head-related transfer function is determined by arranging a pair of speakers 91 and 92 and the listener's mannequin doll in the positional relationship shown in FIG. 6 in the anechoic chamber, and the positions of the left and right ears of the mannequin doll. Is a transfer function between That is, the head-related transfer function is a transfer function between the right speaker and the right ear of the listener, and between the left speaker and the listener's left ear in a state without reverberation.

  Therefore, the binaural processing unit 95 selects a head-related transfer function corresponding to the head position information from a plurality of head-related transfer functions stored in association with the head position information stored therein, The signal is convoluted and the audio signal is synthesized into a binaural signal that sounds as if the sound of the speakers 91 and 92 is heard through headphones.

The crosstalk processing unit 96 performs processing for removing spatial crosstalk in which a sound that should originally reach only one ear reaches the other ear. This spatial crosstalk inhibits the synthesis of binaural signals. Therefore, the crosstalk processing unit 96 transfers the transfer functions G ₉₁ L (ω) and G ₉₁ R (ω) between the left and right ears of the listener from the speaker 91 that change according to the head position information, and the left and right of the listener from the speaker 92. The crosstalk generated by the interaural transfer functions G ₉₂ L (ω) and G ₉₂ R (ω) is canceled. The filter H _i (ω) configuration for canceling the crosstalk is shown in equation (1). i means a speaker number. All transfer functions are complex numbers when described in the frequency domain.

Here, H _R (ω) and H _L (ω) are head-related transfer functions.

  The binaural signal from which the crosstalk component has been canceled is converted into an analog signal by the D / A converter 97, amplified by the amplifier 98, and then supplied to the pair of speakers 91 and 92.

Kenichiro Yamamoto, Ken Naemura, Hiroshi Harashima, “Sound image localization using three-dimensional sensors and transoral processing” Transactions of the Virtual Reality Society of Japan, Vol. 3, No. 3, pp. 981-987, 2000.

However, the conventional method has the first problem that it is necessary to wear the sensor 93 for the purpose of detecting the position of the listener's head, which is troublesome. Further, if the head position moves greatly and the head rotates, for example, at an angle of 60 degrees or more in the direction of the speaker 91 with respect to the front center of the pair of speakers 91 and 92, the listener's ear canal entrance (hereinafter referred to as the ear canal) , The ear hole) becomes a shadow of the listener's pinna when viewed from the speaker 92, and the signal level from the speaker 92 becomes small. At the same time, the zero point at a high frequency also increases rapidly, which causes a second problem that the filter characteristics of the crosstalk canceller become unstable.

  The present invention has been made in view of such a problem, and does not require a sensor to be worn on the listener, and also when the listener moves greatly, the listener's head as in the real world. It is an object of the present invention to provide a trans-oral system that can naturally listen to a sound image having an absolute position that is not affected by the movement of the sound.

  The trans-oral system of the present invention includes three or more speakers, an imaging unit, a face posture analysis unit, a speaker selection unit, a binaural processing unit, a crosstalk processing unit, and a speaker driving unit. Three or more speakers are arranged at a position equidistant from the center of the listener's head, with the sound emission side facing the listener, and two speakers as a set. The imaging unit captures a face image of the listener and outputs face image information. The face posture analysis unit analyzes the posture of the listener's face from the face image information output by the imaging unit, and outputs face posture information. The speaker selection unit receives the face posture information and outputs speaker selection information for selecting a pair of adjacent speakers from the three or more speakers corresponding to the face posture information. The binaural processing unit receives an audio signal, the face posture information, and the speaker selection information as input, and a head related transfer function from the right speaker of the pair of speakers to the right ear of the listener based on the speaker selection information; The head-related transfer function from the left speaker to the left ear is convoluted with the audio signal to output right channel and left channel binaural signals. The crosstalk processing unit receives a binaural signal, face posture information, and speaker selection information, and uses a transfer function from a set of speakers to the right ear of the listener and a transfer function to the left ear based on the speaker selection information. Thus, left and right two-channel speaker drive signals are generated by removing spatial crosstalk components from the left and right two-channel binaural signals. The speaker driving unit receives the speaker selection information and the speaker driving signal, and outputs the speaker driving signal to a set of speakers based on the speaker selection information.

  According to the trans-oral system of the present invention, the posture of the listener's face is obtained by analyzing the facial posture information captured by the imaging unit, so that a sensor attached to the body of the listener as in the prior art is unnecessary. . In addition, there is a pair of speakers that are in a position facing the listener out of three or more speakers corresponding to the posture of the listener's face, that is, the listener's ear hole is not a shadow of the listener's pinna. Therefore, it is possible to provide a trans-oral system that can cope with a large change in the posture of the listener's face.

The figure which shows the function structural example of the trans-oral system 100 of this invention. The figure which shows the function structural example of the face attitude | position analysis part 20. FIG. The figure which shows an example of the speaker selection information of the speaker selection part 30. FIG. The figure which shows the specific example of the speaker drive part. The figure which shows the function structural example of the trans-oral system 200 of this invention. The figure which shows the function structure of the conventional trans-oral system 900.

Embodiments of the present invention will be described below with reference to the drawings. The same reference numerals are given to the same components in a plurality of drawings, and the description will not be repeated.

  FIG. 1 shows a functional configuration example of the trans-oral system 100 of the present invention. Three or more speakers 1 to 6, an image pickup unit 10, a face posture analysis unit 20, a speaker selection unit 30, a binaural processing unit 95, a crosstalk processing unit 96, a D / A converter 97, and an amplifier 98 and a speaker drive unit 40. The binaural processing unit 95, the crosstalk processing unit 96, the D / A converter 97, and the amplifier 98 are the same as the conventional transoral system 900 as can be seen from the reference numerals. In addition, arrangement | positioning of the speakers 1-6 of FIG. 1 is seen planarly centering on a listener's head.

  The three or more speakers 1 to 6 are arranged at a position equidistant from the center of the listener's head with the sound emission side facing the listener. In this example, six speakers 1 to 6 are arranged at equidistant positions with a center angle of 60 degrees around the listener. In stereo reproduction, it is a principle to place two speakers and a listener at the apex of an equilateral triangle (for example, reference document “Introduction to Audio to Enjoy Ultimate Sound”, Seimido Publishing, 1998, p139). In this embodiment, six speakers 1 to 6 are arranged so as to surround the listener. The vertical positions of the speakers 1 to 6 may not be on the same plane. It is only necessary to match the position when the transfer characteristic between the speaker and the listener is measured. The distance between the speaker and the listener depends on the sound pressure level output by the speaker. The distance is approximately in the range of about 50 cm to 5 m.

  The minimum number of speakers as the trans-oral system 100 is arranged in two speakers 1 and 6 arranged in front of the listener as shown by a solid line in FIG. 1, for example, on the left side of the listener. The speaker 2 is composed of three speakers. The third speaker may be the speaker 5 on the right side of the listener. As the number of speakers decreases, the range that can accommodate changes in the listener's posture becomes narrower.

  The imaging unit 10 captures a face image of the listener and outputs face image information. The imaging unit 10 outputs, for example, a face image obtained by capturing a digital camera 16 times per second (16 Hz). The imaging unit 10 may be a digital video camera. A 3D stereo camera may be used.

  The face posture analysis unit 20 analyzes the posture of the listener's face from the face image information and outputs face posture information. Many techniques for analyzing the posture of the listener's face have been studied, and any of these techniques may be used in this embodiment. For example, a plurality of images with different face directions may be taken in advance, and face posture information may be output by determining the degree of coincidence with the images. FIG. 2 shows a functional configuration example of the face posture analysis unit 20 based on the method. The face posture analysis unit 20 includes a face posture determination unit 21 and a face posture data storage unit 22. The face posture data storage unit 22 stores in advance an image of a face for each predetermined angle when the listener's face direction goes around in the horizontal direction. The face posture determination unit 21 compares the current face image information input from the imaging unit 10 with a stored image in which the face direction stored in the face posture data storage unit 22 is known, thereby comparing the current face image information. The face direction closest to is output as face posture information. Alternatively, instead of comparing the image data, the face direction may be directly calculated by recognizing the pattern of the face image data, as is known in face part detection, smile detection, etc. good. The face posture information is given by, for example, an angle representing the orientation of the listener's face.

  The speaker selection unit 30 receives the face posture information and outputs speaker selection information for selecting a pair of adjacent speakers from three or more speakers corresponding to the face posture information. FIG. 3 shows an example of speaker selection information. The angle α in the left column is an angle representing the orientation of the listener's face. The notation R · L in the column of the right speaker represents a binaural signal channel.

  The angle α is defined as α = 0 ° when the listener's face is directed to the center of the speakers 1 and 6, a positive (+) angle in the clockwise direction, and a negative (−) angle in the counterclockwise direction. To do. When the angle α = 0 °, the speaker 1 outputs a left (L) channel binaural signal and the speaker 6 outputs a right (R) channel binaural signal. The speaker selection unit 30 is an encoder that receives face posture information (angle α) as an input.

  Assuming that the listener's face is rotated by +60 degrees (α = 60 °) in the horizontal direction, the listener's left ear hole is hidden behind the pinna of the left ear when viewed from the speaker 1. Then, the filtering processing operation of the crosstalk processing unit 96 becomes unstable. In this case, sound emission from the speaker 1 is blocked (x), and the left (L) channel binaural signal from the speaker 6 and the speaker 5 The speaker is selected so that a right (R) channel binaural signal is output. As a result, both ear holes of the listener are not hidden in the listener's pinna when viewed from the speakers 6 and 5. Accordingly, the filtering processing operation of the crosstalk processing unit 96 operates stably.

  Further, in the case where α = + 120 ° where the listener's face is rotated +60 degrees in the horizontal direction, the speaker is arranged so that the left (L) channel and the right (R) channel of the speaker 5 are output from the speaker 5. Selected. By selecting a pair of adjacent speakers from three or more speakers 1 to 6 corresponding to the face posture information, even if the listener changes the face direction, the absolute position is not affected by the movement. It is possible to naturally provide a sound image having the same.

  As shown in FIG. 3, 0 degrees and −360 degrees, +60 degrees and −300 degrees, +120 degrees and −240 degrees, +180 degrees and −180 degrees, +240 degrees and −120 degrees, +300 degrees and −60 degrees. The combination of speakers in, is the same set. Thus, when six speakers are arranged at a central angle of 60 degrees with the listener as the center, it is possible to provide a natural sound image even if the listener's face is rotated 360 degrees. . For example, when there are three speakers, 6 and 1 and 2, the listener's left and right ear holes provide a natural sound image in a range of +60 degrees to -60 degrees that does not become a shadow of the pinna when viewed from the speakers. be able to.

  The binaural processing unit 95 receives the audio signal, the face posture information, and the speaker selection information as inputs, and the head related transfer function from the right speaker to the right ear of the listener based on the speaker selection information and the left transfer function The head-related transfer function from the speaker to the left ear is convoluted with the audio signal, and the right channel and left channel binaural signals are output. The binaural processing unit 95 stores a plurality of head related transfer functions corresponding to face posture information. The number is, for example, the number of angles stored in the face posture data storage unit 22. Alternatively, when the face posture information is finer than the angle stored by the binaural processing unit 95, for example, once, the head-related transfer function may be obtained by interpolation using an existing interpolation technique. When head related transfer functions having the same angle information are not prepared, the weighted average value of the head related transfer functions on both sides closest to the direction is calculated and used. This idea of interpolating the head-related transfer function can also be applied to the transfer function in the crosstalk processing unit 96.

  The crosstalk processing unit 96 receives a binaural signal, face posture information, and speaker selection information as inputs, and uses a transfer function from a set of speakers to the listener's right ear and a transfer function to the left ear based on the speaker selection information. Thus, left and right two-channel speaker drive signals are generated by removing spatial crosstalk components from the left and right two-channel binaural signals. The crosstalk processing unit 96 performs filtering processing to remove spatial crosstalk components from the left and right two-channel binaural signals using a transfer function from a pair of speakers based on speaker selection information to the right and left ears of the listener. Do. Transfer functions from the two speakers corresponding to the speaker selection information to the right and left ears of the listener are stored in the crosstalk processing unit 96 in advance. The filtering process itself for removing the spatial crosstalk component is the same as the conventional trans-oral system 900.

  The speaker drive unit 40 receives the speaker selection information and the speaker drive signal as inputs, and outputs the speaker drive signal to a set of speakers based on the speaker selection information. The speaker drive unit 40 performs an operation of distributing the left and right two-channel binaural signals shown in FIG. 3 to the speakers 1 to 6.

FIG. 4 shows a more specific functional configuration example of the speaker selection unit 40. The speaker selection unit 40 includes a pair of relays 41 to 46 connected to the speakers 1 to 6, respectively. The pair of relays 41 are relays in which a right (R) channel binaural signal and a left (L) channel binaural signal amplified by an amplifier 98 are respectively connected to one end and the other end is connected to the speaker 1 together. 41 _R and 41 _L are provided. The pair of relays 42 to 46 are also connected to the speakers 2 to 6, respectively, and the configuration thereof is the same as that of the pair of relays 41.

The 0 (−360) output terminal of the speaker selection unit 30 is connected to the control terminals of the relay 41 _L and the relay 46 _R. The 0 (−360) output terminal of the speaker selection unit 30 is “1” when the angle α representing the face direction represented by the face posture information output by the face posture analysis unit 20 is in the range of −60 ° <α <60 °. A selection signal of (logic level 1) is output. When the selection signal of the 0 (−360) output terminal becomes “1”, the relay 41 _L and the relay 46 _R become conductive, the left (L) channel binaural signal is supplied to the speaker 1, and the right ( R) Channel binaural signals are supplied. The supply of binaural signals to the other speakers 2 to 5 is cut off.

The 300 (−60) output terminal of the speaker selection unit 30 is connected to the control terminals of the relay 41 _R and the relay 42 _L. The 300 (−60) output terminal has an angle α of “1” in a range of −120 ° <α ≦ −60 °, and therefore the speaker 1 has a right (R) channel binaural signal and the speaker 2 has a left (L ) The binaural signal of the channel is supplied.

  Table 1 shows the relationship between the angle range of the angle α and the speaker numbers to which the binaural signals of the right (R) channel and the left (L) channel are supplied.

  As shown in Table 1, by selecting a speaker that supplies a binaural signal in correspondence with face posture information (angle α), the listener's ear canal can be heard even if the listener's face turns 360 degrees. It is possible to provide a natural sound image because it is not a shadow of the media.

  The trans-oral system 100 selects two speakers from three or more speakers based on the first feature for obtaining face posture information from the face image of the listener photographed by the imaging unit 10 and the face posture information. It has the 2nd characteristic to do. The trans-oral system of the present invention may have a configuration having only the second feature. The transoral system 200 having such a configuration will be described next.

  FIG. 5 shows a functional configuration example of the trans-oral system 200 of the present invention. The trans-oral system 200 is different in that an input unit 50 is provided instead of the imaging unit 10 and the face posture analysis unit 20 of the trans-oral system 100. The configuration of the other functional units can be basically realized with the same idea.

  The input unit 50 receives the listener's face direction information input from the outside or the head position information indicating the relative position information of the listener's both ears and the speaker, the binaural processing unit 95 and the crosstalk processing unit 96. Output to the speaker selection unit 30. The head position information indicating the listener's face direction information or the relative position information of the listener's both ears and the speaker is a signal having the same meaning as the face posture information described above.

  For example, the user of the trans-oral system 200 may determine the listener's face visually and manually input the listener's head position information with respect to the speaker into the input unit 50. The speaker selection unit 30 is configured to select a set of speakers based on the head position information. The binaural processing unit 95 and the crosstalk processing unit 96 are also configured to select a head-related transfer function and a transfer function based on the head position information. By doing so, similarly to the trans-oral system 100, it is possible to naturally provide a sound image having an absolute position that is not affected by the movement of the listener even if the listener greatly changes the face direction.

  The audio signal input to the binaural processing unit 95 has been described as an example of a digital signal, but the audio signal may be an analog signal. In that case, binaural processing and crosstalk processing are realized by a plurality of analog filters corresponding to face posture information. When the audio signal is given as an analog signal, the D / A converter 97 is not necessary. Further, when the output level of the binaural signal is high, the amplifier 98 is not necessary. Thus, the D / A converter 97 and the amplifier 98 do not characterize the present invention.

In addition, although the speaker selection part 30 which selects a pair of speakers based on face posture information was demonstrated by the example of a functional structure provided independently, the function of the speaker selection part 30 is crossed with the binaural processing part 95. The talk processing unit 96 and the speaker driving unit 40 may be provided respectively. In that case, the speaker selection part 30 is unnecessary. Thus, the trans-oral system of the present invention is not limited to the configuration of the above-described embodiment.
The trans-oral systems 100 and 200 may be configured such that a predetermined program is read into a computer including, for example, a ROM, a RAM, and a CPU, and the CPU executes the program.

  In that case, the program describing the processing contents can be recorded in any computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. More specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape, etc., and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read) Only Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording media, MO (Magneto Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  This program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a recording device of a server computer and transferring the program from the server computer to another computer via a network.

  Each unit may be configured by executing a predetermined program on a computer, or at least a part of the processing contents may be realized as hardware.

Claims (2)

6 or more speakers, each of which is a set of two speakers, with the sound emission side facing the listener at equidistant positions with the center angle being equal to the center of the listener's head;
An imaging unit that captures a face image of the listener and outputs face image information;
A face posture analysis unit that analyzes the posture of the listener's face from the face image information and outputs face posture information;
A speaker selection unit that outputs the speaker selection information for selecting a pair of adjacent speakers from the six or more speakers in correspondence with the face posture information, using the face posture information as an input;
With the audio signal, the face posture information, and the speaker selection information as inputs, the head-related transfer function from the right speaker to the right ear of the listener based on the speaker selection information and the left from the left speaker A binaural processing unit that convolves the head-related transfer function up to the ear with the audio signal and outputs a binaural signal of the right channel and the left channel;
Using the binaural signal, the face posture information, and the speaker selection information as inputs, using a transfer function from a set of speakers to the right ear of the listener and a transfer function to the left ear based on the speaker selection information, A crosstalk processing unit for generating left and right two-channel speaker drive signals obtained by removing spatial crosstalk components from the left and right two-channel binaural signals;
A speaker driving unit that receives the speaker selection information and the speaker driving signal as input and outputs the speaker driving signal to a pair of speakers based on the speaker selection information;
Transoral system comprising 6 or more speakers, each of which is a set of two speakers, with the sound emission side facing the listener at equidistant positions with the center angle being equal to the center of the listener's head;
An input unit for inputting head position information representing a relative direction between the listener's face direction or both ears of the listener and the speaker;
A speaker selection unit that outputs the speaker selection information for selecting a pair of adjacent speakers from among the six or more speakers corresponding to the head position information, with the head position information as an input;
As input an audio signal and the head position information and the speaker selection information, the head-related transfer function from the right speaker of a set of speakers based on the speaker selection information to the right ear of the listener and the left speaker A binaural processing unit that convolves the head-related transfer function from the left ear to the left ear into the audio signal and outputs a binaural signal of the right channel and the left channel;
Using the binaural signal, the head position information, and the speaker selection information as inputs, using a transfer function from a pair of speakers to the right ear of the listener and a transfer function to the left ear based on the speaker selection information A crosstalk processing unit for generating left and right two-channel speaker drive signals obtained by removing spatial crosstalk components from the left and right two-channel binaural signals;
A speaker driving unit that receives the speaker selection information and the speaker driving signal as input and outputs the speaker driving signal to a pair of speakers based on the speaker selection information;
Transoral system comprising

Images