JP3363921B2 - Sound image localization device - Google Patents

Description

Detailed Description of the Invention

[0001]

[Industrial application] In recent years, there has been a demand for human-computer interaction methods such as a man-machine interface and a human interface.
Technologies such as R) and virtual reality (VR) have led to the development of intuitive dialogue methods that utilize the human senses. However, most of the dialog methods that have been developed so far are visual and less auditory.

From the viewpoint of human perception of reality, it is considered that information from the visual sense is the most important, but effective sense of reality cannot be obtained only by the visual sense. This is because human beings recognize the perception of reality by all five senses of information. Considering an interactive device in the future, an audio device capable of localizing a sound image is required. By using this acoustic localization device, the direction of sound and the surrounding environment can be realistically given to humans, and the sense of reality will be improved. For example, by using a head-mounted stereoscopic device called a head mounted display (HMD) as a visual device and an acoustic device capable of localizing a sound image as a sound reproducing device, a virtual world that changes in real time (landscape) You can experience simulation, CAD / CAM, etc.) with computer graphics (CG) and computer sound (CS), and perform intuitive operations. The present invention relates to the acoustic localization device described above,
In particular, the present invention relates to an acoustic localization device that allows a listener to perceive an arbitrary localization of a sound image.

[0003]

2. Description of the Related Art Artificial-reality hearing devices that incorporate three-dimensional sound into a virtual world to express reality include, for example, a system for generating three-dimensional sound using headphones, or A system that generates a stereoscopic sound using a speaker is known. (1) A system that generates stereoscopic sound using headphones. FIG. 9A is a diagram showing the configuration of the above system. In the figure, 91 is a dry source such as a CD as a sound source,
92 is a disk, 93 is a sampler, and 94 is RS232C.
Serial port, 95 is host computer, 96
Is a dedicated computer for three-dimensional sound generation, 97 is headphones, 98
Is a position detecting device, and 99 is a listener. In the figure, the source (dry source) to be output is sampler 9 first.
Record to 3. The host computer 95 is RS-23
Converted to MIDI (Musical Instrument Digital Interface, a digital music signal transmission protocol, hereinafter referred to as MIDI) via the 2C serial port, and selects dry source, selects four output lines, and Controls output timing.

The position and direction of the listener 99 (six degrees of freedom in position and direction) are detected by a position detection device 98, and a dedicated computer 9 for generating a stereophonic sound from a host computer 95.
6. The three-dimensional sound generation dedicated computer 96 converts the dry source into three-dimensional sound based on the position and direction information of the listener 99, and outputs it to the headphones 98 of the listener 99 as left and right sounds. The features of this system are as follows. This is a personal stereophonic sound generation device that uses headphones.
Since headphones are used, there are individual differences in hearing due to the skeleton and hairstyle of the listener, and it is difficult to correct them. Express the dry sauce with a sampler. The listener is given 6 degrees of freedom (position, direction). It is difficult to localize the sound in front of the listener. The video equipment is head mount
It is a system using a display (HMD).

(2) System for Generating Stereoscopic Sound Using Speaker FIG. 9 (b) is a diagram showing the configuration of the above system. In FIG. 9, 101 is a computer capable of generating sound, and 102 is a digital system.・ Surround decoder, 10
3a to 103e are speakers, and 104 is a listener.
In the figure, first, the dry source to be recorded is stored in the memory of the computer 101. Computer 10
1 is digital surround sound from the position of the listener 105 and the sound source (up to 4 types of dry sources can be output simultaneously).
The left and right sounds are generated in consideration of the characteristics of the decoder 102.
The digital surround decoder 102 performs directionality enhancement processing on the generated left and right sounds and a four-direction speaker 103.
Stereo sound is output from a to 103e. The features of this system are as follows. It is a speaker-type three-dimensional sound generation device that can be heard by a plurality of people. Providing a sound field within the speaker placement range. Since stereoscopic sound is expressed by a commercially available digital surround decoder, there is a possibility that it will depend on the directional enhancement processing. The listener is given three degrees of freedom only in position. The video device is a single plane screen.

[0006]

By the way, the system of (1) described above generates stereoscopic sound depending on the position and direction of the experiencer, but since headphones are generally used, localization of sound into space is weak, The movement of the sound image is also such that left and right movement can be expressed. In particular, due to the characteristics of headphones, the sound in the front tends to be localized in the head and the sound in the rear tends to be localized in the higher level,
Localization in all directions is difficult. In addition, in a system using headphones, there are differences in sound recognition due to individual differences in head related transfer functions (skeleton, ear shape, hair, etc.),
It is difficult to express reality.

The system (2) is a system for providing a stereophonic sound to a listener facing the front. The movement of the sound image can be expressed to some extent in the space, but the localization of the sound to the space is weak. Also, in terms of recognition of artificial reality,
There are problems with the degree of freedom of the listener and the dependence on the directionality enhancement processing. The present invention has been made in view of the above-mentioned problems of the prior art, and makes the listener follow the movement of the head of the listener (experiencer) and localize and hear a sound image of an arbitrary sound. It is an object of the present invention to provide a sound image localization device capable of recognizing a sound image with a sense of reality.

[0008]

FIG. 1 is a basic configuration diagram of the present invention. In order to solve the above problems, claim 1 of the present invention
Of the invention, a sound generator 1 for generating an arbitrary sound, and a listener 5
A position detecting device 3 for detecting a listening point such as a position and an angle, a sound control device 2 for generating a sound image, sound reproducing devices 4a, 4b, 4c, 4d for reproducing a sound image, and a virtual sound world. The management computer 6 provided and the place where the sound reproduction device is installed
Collecting arbitrary sounds that occur in different places , the sound generation position
A sound collection device 1 ′ for sending to the sound control device is provided together, the management computer 6 temporally manages the movement or change of any sound based on the output of the position detection device 3, and the sound control device 2 outputs the output of the management computer 6. Based on the information on the position of the listener 5 from the position detection device 3, the arrangement of the sound reproduction devices 4a, 4b, 4c, 4d, and the relation between the sound reproduction device 4a, 4b, 4c, and 4d and the localization position of the generated sound or the collected sound, the generated sound or the collected sound is collected. It is configured to give an effect to the sound and reproduce it.

According to a second aspect of the present invention, in the first aspect of the invention, a movable casing for moving the sound reproducing devices 4a, 4b, 4c, 4d is provided, and the movable casing reduces the recognition of the sound image localization. The sound reproduction device 4 is adapted to the movement of the listener 5 so as to prevent it.
It is configured to move a, 4b, 4c and 4d. The invention of claim 3 of the present invention is the invention of claim 1 or contract.
In the invention of claim 2 , the image generation device 7 for generating an image synchronized with a sound is provided, and the image generation device 7 is the sound reproduction device 4.
The sound image output by a, 4b, 4c, and 4d and the image 8 synchronized with the movement of the listener 5 are provided to the listener 5.

[0010]

In the first aspect of the present invention, the position of the head of the listener 5 is detected by the position detecting device 3 and given to the sound control device 2. The management computer 6 temporally manages the movement or change of an arbitrary sound based on the output of the position detection device 3 .
It The sound collector 1'is different from the place where the sound reproduction device is installed.
Collect any sound that is generated in
Both are sent to the sound control device , and the sound control device 2 obtains the positional relationship between the listener's listening point and the sound reproduction devices 4a to 4d from the position and direction of the head of the listener 5 detected by the position detection device 3,
By setting this and an arbitrary sound generation position, the sound information generated by the sound generation device 1 is added to the sound information based on the positional relationship between the three parties.
The effects such as control of the sound pressure ratio, delay of sound information, frequency conversion, etc. are given and output to the sound reproduction devices 4a, 4b, 4c, 4d. Since the sound image is localized by following the position and direction of the listener's head, the listener can recognize the sound image localization with a sense of reality. Further, since the sound collecting device 1 ′ captures the sound generated at the remote place and the sound control device 2 gives an effect to the collected sound and outputs the sound to the sound reproducing devices 4a, 4b, 4c, 4d, the remote place. The sound of can be experienced by the listener.

According to the second aspect of the present invention, the sound reproducing devices 4a, 4b, 4c and 4d are moved in accordance with the movement of the listener 5 so as to prevent the sound image localization from being deteriorated. It is possible to prevent a reduction in the reproduction performance of the reproduction device and a reduction in the recognition of the sound image localization due to the movement of the listener. According to the third aspect of the present invention, there is provided the image generation device 7 for generating the image synchronized with the sound, and the image generation device 7 outputs the sound images output by the sound reproduction devices 4a, 4b, 4c, 4d and the movement of the listener 5. Since the image 8 synchronized with the above is provided to the listener 5, the image can be added to an arbitrary sound and the sense of reality can be further enhanced.

[0012]

FIG. 2 is a diagram showing a first embodiment of the present invention, in which 21 is sound information (dry source).
, 22 is a sound control device that exerts an effect on the sound information generated by the sound generation device 21, 23 is a position detection device that detects the position of the listener's head (6 degrees of freedom in position and direction), Reference numerals 24a to 24d are sound reproducing devices including, for example, speakers, and 25 is a listener. In FIG. 2, the sound reproduction devices 24a to 24d are arranged so as to surround the listener 25, and the position of the head of the listener 25 is determined by the position detection device 23.
Is detected by the sound control device 22 and given to the sound control device 22. The sound control device 22 controls the listener 25 detected by the position detection device 23.
From the position and direction of the head of the listener, the listening point of the listener and the sound reproducing device 24a
To 24d, the position of generation of an arbitrary sound is set, and based on the positional relationship of these three, the sound information generated by the sound generator 21 is effective.

As an effect to be given to the sound information, for example, when the sound information generated from the sound generating device 21 is allocated to the sound reproducing devices 24a to 24d, the sound pressure ratio is changed, the sound information is delayed, or the frequency is changed. Or convert. In addition, the reflected sound and reverberant sound are calculated to localize the sound image, and the listener recognizes the size of the room. In the present embodiment, as described above, since the sound image is localized by following the position and direction of the listener's head,
The listener can recognize the sound image with a sense of reality.

FIG. 3 is a diagram showing a second embodiment of the present invention, in which the same components as those shown in FIG. 2 are designated by the same reference numerals, and in this embodiment, FIG. Instead of the sound generating device 21 shown, a sound collecting device 21 'provided at a remote place is provided, and the other configurations are the same as those of the first embodiment. In FIG. 3, the sound collecting device 21 ′ collects the sound in the space apart from the listener 25 and sends it to the sound control device 22. At that time, if necessary, the sound collecting device 21 'sends information on the generation position of the sound to be collected. The sound control device 22 applies the same effect to the sound information sent from the sound collecting device 21 ′ as described in the first embodiment, and the sound reproducing devices 24 a through 24 a.
The sound image is localized at d. In the present embodiment, as described above, the listener can experience the sound of a remote place with a sense of reality.

FIG. 4 is a diagram showing a third embodiment of the present invention, in which the same components as those shown in FIG. 2 are designated by the same reference numerals, and in this embodiment, a sound reproducing device 24a is used. Through 24
d is attached to the movable housings 29a to 29d so that it can be constantly directed toward the listener 25. In the figure, as in the first embodiment, the sound control device 22 gives an effect to the sound generated by the sound generation device 21 and gives it to each of the sound reproduction devices 24a to 24d, and the listener detected by the position detection device 23. Based on the position information of 25, the movable housing movement information for constantly orienting the sound reproducing devices 24a to 24d toward the listener 25 is given. In the present embodiment, as described above, the movable casings 29a to 29a are used.
Since the sound reproduction devices 24a to 24d are directed to the listener 25 by d, it is possible to prevent the reproduction performance of the sound reproduction device from being insufficient and the reduction in the recognition of the sound image localization due to the movement of the listener.

FIG. 5 is a diagram showing a fourth embodiment of the present invention. The same parts as those shown in FIG. 2 are designated by the same reference numerals, and this embodiment is the same as that of FIG. A management computer 26 for managing arbitrary sounds is provided in the second embodiment, and other configurations are the same as those in the first embodiment. In FIG. 5, the management computer 26 is a computer that manages the movement of any sound and the change of sound (may include a time component). For example,
When making the listener 25 recognize the sound of the bees flying around the listener 25, the management computer 26 causes the sound generation device 21 to emit the sound of the bees flying and causes the sound control device 22 to operate while the bees are flying. Conveys bee location information. The sound control device 22 is
Similar to the first embodiment, the sound generated by the sound generation device 21 is assigned to each of the reproduction devices 24a to 24d based on the position information provided from the management control device 26, and the listener 25 is made to recognize the sound of a bee flying. In the present embodiment, as described above, since the management computer 26 is provided to manage the movement and change of any sound, it is possible to make the listener 25 recognize the sound more realistically.

FIG. 6 is a diagram showing a fifth embodiment of the present invention. The same parts as those shown in FIG. 2 are designated by the same reference numerals, and this embodiment is the same as that of FIG. Video playback device 2
7 and screens 28a to 28d for displaying images
Is provided, and the other structure is the same as that of the first embodiment. In FIG. 6, the image reproducing device 27 generates an image that gives the experience person (listener) a sense of reality based on the position information of the object generating the sound and causes the experience person (listener) to experience. As described above, in the present embodiment, the video playback device 27 and the screens 28a to 2 for displaying the video are used.
Since 8d is provided, it is possible to add an image to an arbitrary sound and enhance the sense of reality.

FIG. 8 is a diagram showing a specific embodiment of the present invention. This embodiment is the same as the fourth embodiment shown in FIG. 5 and the embodiment shown in FIG. It shows an embodiment in which a stereoscopic screen display device by (HMD) and a real-time image generation device are provided. A head-mounted display (HMD) is a device that enables stereoscopic viewing by allowing two liquid crystal display devices in front of the eye to be individually viewed by the left and right eyes by an optical system. FIG. 7 is a diagram showing the arrangement of speakers in the embodiment shown in FIG.
L and 50R are left and right speakers provided on the front surface of the experience person (listener) 25, 51L and 51R are left and right speakers provided on the side surface of the experience person (listener) 25, 52
L and 52R are left and right speakers provided on the back surface of the experience (listener) 25.

In FIG. 8, the same components as those shown in the embodiment of FIGS. 5 and 6 are designated by the same reference numerals, and in FIG. 8, 23 is the position of the head of the experience person (listener). A position detection device that detects (6 degrees of freedom in position and direction), 25 is an experience person (listener), 27 is a video generation device that generates a stereoscopic image in real time, and 28 'is a stereoscopic image generated by the video generation device 27. A screen, 29 is a sound control device for generating a control signal for giving an effect to a sound, and outputs a signal for controlling a digital sampler, a programmable line selector, etc. described below. Further, 31 is a digital sampler for generating a programmed sound at a MIDI timing, 32 is a programmable line selector for allocating the sound generated by the digital sampler 31 to a digital sound processor, and 33a to 33c are programmable line selectors. Digital sound that performs sound pressure, delay, and frequency conversion on the input sound source from the line selector 32.
It is a processor.

34a to 34f are amplifiers for amplifying the outputs of the digital sound processors 33a to 33c, 35a to 35c are digital mixing processors for controlling the sound pressure of the outputs of the amplifiers 34a to 34f and the output of the DAT 38, and 36a to 36c. 36f amplifies the outputs of the digital mixing processors 35a to 35c to amplify the outputs of the experience (listener) 25 shown in FIG. 7 to the left, right, left and right speakers 50L, 50R, 51L, 51R, respectively.
An amplifier for outputting to 52L and 52R, and 38 is a DAT (digital audio tape) on which sound is recorded. Four
0, 41, 42 are RS-232C / MIDI serial
Port, sound controller and digital sampler 31,
Control signals are transmitted between the programmable line selector 32, the digital sound processors 33a to 33c, and the digital mixing processors 35a to 35c. In addition, 43, 44 and 45 are RS232C
Is a MIDI expander for converting a serial signal of the above into MIDI.

In FIG. 8, the position and direction of the head of the experiencer (listener) 25 are detected by the position detection device 23 and given to the management computer 26. Management computer 26 is an experienced person (listener)
Based on the position and direction signals of the head of No. 25, the kind of sound to be generated and sound position information are given to the sound control device 29 and displayed on the video generation device 27 as in the fourth embodiment of FIG. And the position information of the experience person (listener) 25 is given. Based on the output of the management computer 26, the video generation device 27 displays the position of the object in the virtual world in real time according to the motion of the experiencer (listener) 25 in a head mounted display (H).
It is displayed on the left and right of the MD) and provides the experience (listener) 25 with a stereoscopic image of computer graphics. The sound control device 29 uses the position information of the sound output by the management computer 26 to transmit a digital sampler 31, a programmable line selector 32, and digital sound processors 33a-3a through an RS-232C / MIDI serial port.
3c and digital mixing processor 35a-
35c is controlled to generate a three-dimensional sound.

The digital sampler 31 outputs the programmed sound at the timing of the MIDI output from the sound control device 29 and gives it to the programmable line selector 32. The programmable line selector 32 allocates the sound generated by the digital sampler 31 to the digital sound processors 33a to 33c based on the control signal of the sound control device 29. Digital sound
The processors 33a to 33c perform sound pressure, delay, and frequency conversion of sounds on the front, side, and back of the experience (listener) 25 with respect to the allocated input sound source. The outputs of the digital sound processors 33a to 33c are amplifiers 34a to
It is amplified by 34f and given to the digital mixing processors 35a to 35c. The digital mixing processors 35a to 35c are amplifiers 34a to 34f.
Of the speaker 50L, 50R, 51L, 51 shown in FIG. 7 via the amplifiers 36a to 36f by mixing the output of the DAT and the output of the DAT 37 based on the control signal of the sound control device 29.
It outputs to R, 52L, 52R, and provides a stereoscopic sound to the experience person (listener) 25.

As described above, in this embodiment, the experience (listener) is detected based on the positional relationship between the experience (listener) and the object in the virtual world by detecting the head movement of the experience (listener). Since the stereoscopic image and the stereoscopic sound of the virtual world that match the movement of the virtual world are provided, the experience person (listener) can perceive the virtual world as reality and improve the operability of the virtual world.

[0024]

As is apparent from the above description,
In the present invention, the position, direction, etc. of the head of the experiencer (listener) is detected, and the relationship between the position of the listener, the position of the sound reproduction device, and the localization position of the generated sound is arbitrarily set. Since the generated sound is reproduced by giving an effect, it is possible to localize the sound image by the interaction between the human and the computer, and it is possible to enhance the sense of reality and make the experience person (listener) recognize the sound. In particular, by combining with an image generation device, it can be applied to various fields such as CAD / CAM, tele presence, tele presence, education, and architectural entertainment, and the effect is extremely large.

[Brief description of drawings]

FIG. 1 is a basic configuration diagram of the present invention.

FIG. 2 is a diagram showing a first embodiment of the present invention.

FIG. 3 is a diagram showing a second embodiment of the present invention.

FIG. 4 is a diagram showing a third embodiment of the present invention.

FIG. 5 is a diagram showing a fourth embodiment of the present invention.

FIG. 6 is a diagram showing a fifth embodiment of the present invention.

FIG. 7 is a diagram showing an arrangement of a reproducing apparatus according to a specific embodiment of the present invention.

FIG. 8 is a diagram showing a specific example of the present invention.

FIG. 9 is a diagram showing a conventional example.

[Explanation of symbols]

1, 21 Sound generating device 1 ', 21' Sound collecting device 2, 22 Sound control device 3, 23 Position detecting device 4a, 4b, 4c, 4d, 24a, 24b, 24c, 2
4d, 50L, 50R, 51L, 51R, 52L, 52
R sound reproduction device 5,25 listener 6,26 management computer 7,27 video generation device 29a, 29b, 29c, 29d movable housing 31 digital sampler 32 programmable line selector 33a, 33b, 33c digital sound processor 34a, 34b, 34c, 34d, 34e, 34f, 3
6a, 36b, 36d, 36e, 36f Amplifiers 35a, 35b, 35c Digital mixing processor 38 Digital audio tape (DAT)

─────────────────────────────────────────────────── ─── Continuation of the front page (56) Reference JP-A-1-279700 (JP, A) JP-A-1-239674 (JP, A) JP-A-52-30402 (JP, A) JP-A-4- 192066 (JP, A) Actual Kaihei 3-105099 (JP, U) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04S 7/00 H04S 1/00

Claims (3)

(57) [Claims] 1. A sound generation device for generating an arbitrary sound, a position detection device for detecting a listening point such as a position and an angle of a listener, a sound control device for generating a sound image, and a sound reproduction device for reproducing the sound image. , Occurs in a place different from the place where the management computer that provides the virtual sound world and the sound playback device are installed
Collect any sound, and send it to the sound control device together with the position of the sound.
A sound collecting device is provided, the management computer temporally manages the movement or change of any sound based on the output of the position detection device, and the sound control device relates to the position of the listener from the position detection device based on the output of the management computer. A sound image localization device characterized in that an effect is exerted on a generated sound or a collected sound depending on the relationship between the information, the arrangement of the sound reproduction device, and the localization position of the generated sound or the collected sound which is arbitrarily set. 2. The sound reproducing device is provided with a movable casing, and the movable casing moves the sound reproducing device in accordance with the movement of the listener so as to prevent deterioration of recognition of sound image localization. 1. Sound image localization device. 3. An image generating apparatus for generating an image synchronized with sound, wherein the image generating apparatus provides the listener with an image synchronized with the sound image output by the sound reproducing apparatus and the movement of the listener. The sound image localization apparatus according to claim 1 or 2 .