JP2018152834A - Method and apparatus for controlling audio signal output in virtual auditory environment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and an apparatus for controlling an audio signal output capable of causing a listener to obtain a real virtual auditory experience.SOLUTION: An apparatus for controlling audio signal output in a virtual auditory environment includes: audio input means arranged to receive one or more audio signals corresponding to one or more sound sources in a virtual auditory environment; azimuth information acquisition means arranged to acquire azimuth information of an object in a virtual auditory environment; identification means that is disposed for identifying a collection of one or more pieces of sound source directional data corresponding to directional information respectively representing transmission characteristics with which one or more sound sources transmit sound waves in directions corresponding to the directional information in a virtual auditory environment; processing means arranged to respectively process one or more sound signals on the basis of a collection of one or more sound source orientation data to obtain a binaural sound signal; and sound output means arranged to output the binaural sound signal.SELECTED DRAWING: Figure 1

Description

  Embodiments of the present invention relate to virtual auditory technology, and more particularly to a method and apparatus for controlling audio signal output in a virtual auditory environment.

  Human hearing is another important route for sensing external information other than vision. In a real auditory environment, sound waves radiated from a sound source are transmitted to the listener via direct and environmental reflection routes, and after being scattered or reflected by physiological structures such as the listener's head, pinna, and trunk. Since it is transmitted to both ears, the sound pressure signal received by both ears includes the sound space information and the acoustic space information of the environment. The auditory system (including the higher nervous system) uses this information to produce a corresponding spatial hearing and a subjective sense of the sound source positioning and the surrounding acoustic environment.

  The virtual auditory environment is generated artificially or the acoustic environment is controlled so that a listener feels a sense of being placed in a natural acoustic environment. Since the binaural audio signal contains the main information of the audio, the listener senses the subjective sensation as if he was placed in a specific acoustic environment by artificially simulating the binaural audio signal and reproducing it with an earphone or speaker. Can occur.

  In the prior art, several methods have been disclosed in which a listener can virtually imagine a corresponding spatial hearing by an earphone or a speaker. For example, Patent Document 1 provides a signal processing method for 5.1-channel surround earphone playback. By using this method, it is possible to eliminate the intra-head positioning effect of the earphone playback, and a room in a listening room. A natural 5.1ch surround effect can be reproduced on the premise that it is not necessary to simulate the reflected sound.

  In these current schemes, earphones or speakers only provide the listener with a natural spatial auditory effect. However, when the listener himself / herself moves or rotates, the sound field that he / she feels does not change accordingly, and a more realistic immersion effect cannot be obtained.

China Patent Application Publication No. CN1402593

  The problem to be solved by the present invention is that a virtual auditory environment is constructed, and a listener who is placed on the virtual auditory environment hears a sound effect adapted to the physical orientation according to the difference in his / her physical orientation, thereby listening. Allows the listener to obtain a realistic virtual auditory experience, and further adjusts the virtual auditory environment according to the change in the physical orientation of the listener so that the listener can feel the sound effect adapted to the change in the physical orientation. Accordingly, it is an object of the present invention to provide a method and apparatus for controlling audio signal output that can enhance the virtual auditory experience of a listener.

  According to an embodiment of the present invention, a method for controlling audio signal output in a virtual auditory environment includes a first acquisition step of acquiring azimuth information of an object in the virtual auditory environment, and 1 in the virtual auditory environment for the object. A first specifying step of specifying a set of one or more sound source azimuth data corresponding to the azimuth information, each of which represents transmission characteristics in which one or a plurality of sound sources transmit sound waves in a direction corresponding to the azimuth information; First processing for processing one or a plurality of audio signals corresponding to the one or the plurality of sound sources based on the set of the one or more sound source azimuth data so as to obtain a binaural sound signal for performing Steps.

  In a further embodiment, the method includes a second acquisition step of acquiring azimuth information after the change of the object in the virtual auditory environment, and one or more sound sources in the virtual auditory environment for the object. Second specification for specifying another one or another set of sound source azimuth data corresponding to the changed azimuth information, each representing transmission characteristics for transmitting a sound wave in the azimuth corresponding to the changed azimuth information One or more sounds corresponding to the one or more sound sources based on the set of one or more other sound source orientation data to obtain a step and a binaural sound signal to output A second processing step for processing each of the signals.

  An apparatus for controlling audio signal output in a virtual auditory environment according to an embodiment of the present invention is arranged to receive one or more audio signals corresponding to one or more sound sources in the virtual auditory environment. Input means, orientation information acquisition means arranged to acquire the orientation information of the object in the virtual auditory environment, and one or more sound sources in the virtual auditory environment for the object Identifying means arranged to identify a set of one or more sound source azimuth data corresponding to the azimuth information, each representing a transmission characteristic transmitted in a azimuth corresponding to, and so as to obtain a binaural audio signal Processing means arranged to process each of the one or more audio signals based on a set of one or more sound source orientation data; Serial and an audio output means arranged to output a binaural audio signal.

  In a further embodiment, the azimuth information acquisition means is further arranged to acquire azimuth information after the change of the object in the virtual auditory environment, and the specifying means further includes the virtual auditory sense for the object. One or more other sound source orientations corresponding to the changed orientation information, each representing a transmission characteristic in which one or more sound sources in the environment transmit sound waves in the orientation corresponding to the changed orientation information Arranged to identify a set of data, and the processing means is further adapted to obtain the binaural audio signal based on the one or more other sets of sound source azimuth data. It arrange | positions so that an audio | voice signal may be processed, respectively.

1 is a block diagram illustrating a configuration of an apparatus for controlling audio signal output in a single object virtual auditory environment according to an embodiment of the present invention. 1 is a schematic diagram of a single object virtual reality system according to an embodiment of the present invention. FIG. 5 is a flowchart of a method for controlling audio signal output in a single object virtual auditory environment according to an embodiment of the present invention. 1 is a block diagram of an apparatus for controlling audio signal output in a multi-object virtual auditory environment according to an embodiment of the present invention. 1 is a schematic diagram of a multiple object virtual reality system according to an embodiment of the present invention. 4 is a flowchart of a method for controlling audio signal output in a multi-object virtual auditory environment according to an embodiment of the present invention.

  A transmission characteristic in which a sound source transmits sound waves in a specific direction can be represented as a set of function data. A set of function data indicating such sound wave transmission characteristics indicates that a sound signal is transmitted from a sound source. It can be used to process audio signals so as to represent transmission characteristics transmitted in a direction. When such a processed audio signal is converted into audio by a playback device, the audio is transmitted in such a manner that the sound source transmits sound waves in that direction so that the listener can feel the virtual sound source space direction. Represents. Each sound source represents a transmission characteristic in which a sound source transmits sound waves in a plurality of specific directions using a set of a plurality of function data, and each sound signal is processed using each of the plurality of function data sets. Can be represented in the audio signal, respectively. A virtual auditory environment can be constructed by such a plan, and based on this, the physical physical orientation of the listener is reflected in a specific orientation in the virtual auditory environment, and the actual physical orientation and virtual of the listener are different. If a correspondence is established between different specific orientations in the auditory environment, the listener can hear a sound effect that matches the physical orientation according to the difference in the physical orientation of the listener. With such a technical proposal submitted to the present invention, the sound perceived by the listener does not express the predetermined spatial orientation sense, and the spatial orientation sense that matches the physical orientation of the listener. Can be expressed.

  In the technical solution of the present invention, the set of function data for representing the sound wave transmission characteristics is referred to as a set of sound source direction data. The set of sound source direction data applied to the present invention includes, for example, a set of head related transfer function data (HRTF), a set of interaural time difference (ITD) data, and binaural Any suitable set of data that can represent the transmission characteristics of sound waves emitted from a sound source transmitted in a certain direction, such as a set of intensity difference data (IID) between them.

  ITD refers to the time difference when a sound signal reaches both ears due to the difference in distance from the sound source to the left and right ears. IID refers to an intensity difference when a sound signal reaches both ears due to a difference in distance from the sound source to the left and right ears. ITD and IID are both functions of sound source position and sound wave frequency. If the set of sound source positioning data is a set of ITD and IID data, the user can identify whether the sound source is located on the left side or the right side thereof.

  HRTF is an acoustic transfer function from a sound source to both ears in the case of a free sound field, and a characteristic change that occurs when a sound wave emitted from a sound source in a free sound field is incident on a point in the ear canal at a certain angle. Used to represent. HRTF is a function of sound source position, sound wave frequency, and human body surface shape and properties. The unit impulse response from the sound source to the human body measurement point is referred to as a head related impulse response (HRIR). HRTF is the Fourier transform of HRIR. If the set of sound source positioning data is a set of HRTF data, the user can determine whether the sound source is located in front of it, located behind, located above, located below, located on the left side, Or it can be identified whether it is located on the right side.

  In order to better understand the present invention, an application of a set of sound source azimuth data in the technical solution of the present invention will be described below by taking a set of HRTF data as an example. Before describing each embodiment of the present invention in detail, first, an HRTF acquisition method will be briefly introduced.

  The HRTF can be obtained by experimental measurements. Measurements are usually made in a free sound field (eg, a muffler room). The measurement principle of HRTF is similar to the measurement of normal acoustic transfer function. A sound source (for example, a small speaker system) generates an excitation signal, picks up a sound pressure signal in both ears of the specimen using a pair of microphones, and then obtains a transfer function by a signal processing method. By changing the relative position between the sound source and the specimen, HRTFs at different sound source positions can be obtained. For picking up sound pressure signals in both ears, for a human head model specimen, a microphone can be placed at the end of the ear canal simulator, which corresponds to measuring at the location of the eardrum. For human specimens, an ultra-small microphone can be placed in the closed ear canal for measurement.

  The HRTF can also be obtained by a calculation method. For example, the HRTF can be calculated using the Raylei scattering formula, in which the head is simplified to a rigid sphere and both ears are simplified to two opposing points on the sphere, and the rigid sphere impinges a wave on the plane. For example, when the influence of the trunk is taken into consideration, the head and the trunk can be simplified to two spheres having different radii, and the HRTF can be calculated using the Green function and the multiple scattering method. It is also possible to convert the outline of the research object into a computer image using a method such as laser or CT scan, and calculate the HRTF using an approximation method.

  The above is only a few known methods for acquiring HRTFs, but any HRTF acquired by any known or possibly developed method applies to the basic idea of the present invention. Therefore, it should be understood that it falls within the protection scope of the present invention.

  In the following, embodiments of the present invention will be described for a single object virtual auditory environment and a multi-object virtual auditory environment, respectively, with reference to the drawings. The virtual auditory environment can be applied to various scenes such as virtual aerospace training, virtual military training, virtual entertainment system, and virtual conference system. In a normal case, the various application scenes listed above include a virtual auditory environment and a virtual visual environment at the same time, and both cooperate with each other to construct a relatively complete virtual reality environment. In the virtual reality environment, the user is reflected in the object in the virtual reality environment as an experience person, and the user's action is correspondingly reflected in the activity of the object in the virtual reality environment. The movement of the object in the real scene can be realized. In the latter part, for convenience of understanding and description, it is necessary to explain that the user in the real world and the object in which the user in the virtual reality environment is not strictly distinguished. For example, if it is necessary to acquire the orientation information of the object in the virtual reality environment, the actual operation step is performed by detecting the physical physical orientation of the user in the real world by the orientation detection means attached to the user, and then the user in the real world. That is, the orientation information of the object in the virtual reality environment is acquired based on the correspondence relationship with the object in the virtual reality environment. For convenience of understanding and description, this type of operation can be referred to as directly detecting object orientation information. It is also necessary to explain that the virtual auditory environment described in the present invention can exist independently without depending on the virtual visual environment.

Single Object Virtual Auditory Environment FIG. 1 shows an apparatus 10 (hereinafter abbreviated as “apparatus 10”) for controlling audio signal output in a single object virtual auditory environment according to an embodiment of the present invention. In a single object virtual auditory environment, there is only one object. As shown in FIG. 1, the apparatus 10 includes a voice input unit 110, a direction information acquisition unit 120, a specifying unit 130, a processing unit 140, and a voice output unit 150.

  Referring to FIG. 1, voice input means 110 is for receiving a voice signal and transmitting it to processing means 140. The audio signal may be, for example, an audio signal collected by a microphone or an audio signal output from an audio player. The audio signal may be, for example, a monaural, 2-channel or stereo audio signal.

  In order to specify the orientation of the object in the virtual auditory environment, the orientation information acquisition unit 120 acquires the orientation information of the object. The orientation information includes the orientation of the object in the single object virtual auditory environment.

  The azimuth information of the object can be provided to the azimuth information acquisition means 120 by the azimuth information provision means (not shown). The function of the direction information providing means can be realized by various methods, and these methods may be used alone or in combination. For example, the user can be equipped with an orientation detection means, and the orientation detection means can detect the orientation of the object shown in the virtual auditory environment and provide the detected orientation information to the orientation information acquisition means 120. Alternatively, the user may be provided with a user input interface, and the user input interface may receive an input about the direction of the object imaged in the virtual auditory environment and provide the direction information to the direction information acquisition unit 120. The orientation detection means may be any suitable element and combination of elements used to detect object orientation, such as a gyro, compass, accelerometer or the like. The input of the user input interface may include various methods such as clicking, dragging on characters, symbols, sounds, or icons in the interface.

  The azimuth information acquisition unit 120 acquires the azimuth information of the object and then transmits it to the identification unit 130. Based on the direction information of the object, the specifying unit 130 specifies a set of one or a plurality of HRTF data corresponding to the direction information for the object, and transmits the specific information to the processing unit 140. The specific information may be in various formats, for example, the specific information may include the set of one or more HRTF data itself, or only the index of the specified set of one or more HRTF data. May be included. Regarding the latter, the processing means 140 needs to store a set of HRTF data corresponding to each index.

  The set of one or more HRTF data specified for the object described above represents a transmission characteristic in which one or more sound sources in a single object virtual auditory environment transmit sound waves in the direction in which the object exists. Used for. When the sound source for the object is a single sound source in the single object virtual auditory environment, the specifying unit 130 specifies a set of one HRTF data corresponding to the single sound source only for the object. When the sound source for the object is a plurality of sound sources in the single object virtual auditory environment, the specifying unit 130 specifies a set of a plurality of HRTF data corresponding to the plurality of sound sources for the object.

  A set of HRTF data for representing transmission characteristics in which a sound source in a virtual auditory environment transmits sound waves in the direction in which the object exists can be acquired by an experimental measurement method or a calculation method. In one example, a set of azimuth information and HRTF data corresponding to each other can be acquired in advance by an experimental measurement or calculation method. The obtained correspondence relationship may be stored in advance in a storage unit (not shown) inside the apparatus 10 or may be stored in advance in an external memory outside the apparatus 10. In this example, the identifying means 130 is based on the orientation information of the object and the correspondence relationship between the orientation information stored in advance and the set of HRTF data, and the object has one or more HRTFs that match the orientation information. A set of data can be selected.

  In actual measurement of HRTF, normally only M finite spatial orientations can be measured, but it should be understood that all spatial orientations cannot be measured. If the orientation information transmitted to the means 130 is not included in the M finite spatial orientations, the identifying means 130 uses a spatial interpolation method, for example, a linear interpolation method, from a set of M known HRTF data, A set of HRTF data corresponding to the orientation information can also be calculated.

  In another example, it is necessary when the specifying unit 130 calculates the orientation information of the object and the prestored HRTF without acquiring and storing the correspondence relationship between the orientation information and the set of HRTF data in advance. Based on other related information, a set of one or more HRTF data corresponding to the orientation information can be calculated for the object in real time.

  Based on the set of one or more HRTF data corresponding to the orientation information identified by the identifying means 130 for the object, the processing means 140 is one in the single object virtual auditory environment from the speech input means 110. Alternatively, one or a plurality of audio signals corresponding to a plurality of sound sources are processed to obtain a binaural audio signal for output, and the binaural audio signal is transmitted to the audio output means 150. Of these, two groups of HRTF data for the left and right ears are included in each set of HRTF data corresponding to the orientation information specified for the object.

  Specifically, for each sound source in the virtual auditory environment, the processing means 140 first performs a Fourier transform on the sound signal corresponding to the sound source to obtain a frequency domain display of the sound signal. Next, the processing means 140 multiplies the two groups of HRTF data corresponding to the left and right ears with the frequency domain display of the audio signal, respectively, and then performs inverse Fourier transform so as to obtain a binaural audio signal to be output. Do.

  The above corresponds to the signal processing method in the frequency domain, but in the case of the time domain, output is performed by convolving two groups of HRIR data corresponding to the left and right ears and the audio signal. It can be understood that a binaural audio signal is obtained.

  The audio output means 150 provides a binaural audio signal to the user.

  In view of the possibility that the orientation of an object in a single-object virtual auditory environment may change, in a further embodiment, the device 10 may determine a post-change for that object based on the changed orientation information of the object. By specifying another or a plurality of HRTF data sets corresponding to the azimuth information, it is possible to realize an effect of selecting, in real time, a set of HRTF data matching the current azimuth for the object. it can. In the following, this further embodiment will be described in detail.

  When the orientation of the object in the virtual auditory environment changes, the orientation information providing means (not shown) can acquire the orientation change of the object and provide the orientation change information to the orientation information acquisition means 120. The function of the direction information providing means can be realized by various methods. For example, the orientation detection means detects the orientation change of the object reflected in the virtual auditory environment, and / or the user input interface receives the input about the orientation change of the object reflected in the virtual auditory environment, Direction change information can be provided to the direction information acquisition means 120. For example, the azimuth detecting means can detect the azimuth change of the object in real time or periodically, and can detect the azimuth change of the object in response to the trigger. On the other hand, the orientation change of the object can be input at any time by the user input interface. The orientation change of the object in the single object virtual auditory environment may include horizontal rotation, translation, vertical rotation, translation, or any combination thereof.

  The azimuth information acquisition unit 120 calculates azimuth information after the change of the object based on the original azimuth information of the object and the azimuth change information acquired this time after acquiring the azimuth change information from the azimuth information providing unit (not shown). At the same time, the direction information after the change is provided to the specifying unit 130. Based on the azimuth information after the change of the object, the specifying unit 130 specifies one or another set of HRTF data corresponding to the changed azimuth information for the object.

  Alternatively, the azimuth information providing unit can directly acquire the azimuth after the change of the object, and can also provide the azimuth information acquisition unit 120 with the changed azimuth information. In this case, the azimuth information acquisition unit 120 does not need to execute the above-described calculation, and only needs to transmit the acquired azimuth information after change to the identification unit 130. Based on the above, another one or another set of HRTF data corresponding to the changed orientation information is specified for the object.

  The identification means 130 identifies another one or another set of HRTF data corresponding to the changed azimuth information for the object. The identification means 130 described in the preceding sentence describes the azimuth information for the object. Is similar to the method of identifying one or more sets of HRTF data corresponding to, and will not be repeated here for the sake of brevity.

  After the identifying means 130 identifies another set of one or more HRTF data corresponding to the changed azimuth information, the processing means 140 inputs speech based on the one or more sets of HRTF data. One or more audio signals corresponding to one or more sound sources in the virtual auditory environment from means 110 are processed and a binaural audio signal is obtained for output. Thereafter, the audio output means 150 provides the user with a binaural audio signal.

  The manner in which the processing means 140 processes one or more audio signals based on another set of one or more HRTF data is the same as the processing means 140 described in the preamble of one or more HRTF data. It is similar to a scheme for processing one or more audio signals based on a set and will not be repeated here for the sake of brevity.

  In the latter part, taking a single object virtual auditory environment for simulating a beach scene as an example, FIG. 2 will be combined to further explain the use of the apparatus for controlling the audio signal output of FIG. 1 in a specific application scene.

FIG. 2 illustrates an exemplary single object virtual reality system 20 according to an embodiment of the present invention. As shown in FIG. 2, the single object virtual reality system 20 includes an audio / video player 210, an apparatus 10 (hereinafter abbreviated as “apparatus 10”) for controlling the audio signal output of FIG. It includes binaural earphones 230 and virtual reality glasses 240.
Referring to FIG. 2, the audio / video player 210 is for providing an audio signal and a video signal. The audio / video player 210 may be any suitable device having an audio / video reproduction function, such as a personal computer, a smartphone, a personal digital assistant, or an MP3 player. In the beach scene of the present embodiment, when the user wears the binaural earphone 230 and the virtual reality glasses 240, the audio / video player 210 displays the virtual reality so as to present the virtual visual environment of the blue sea and the white sand beach to the user. The video signal is transmitted to the glasses 240, and the audio signal processed by the device 10 is transmitted to the binaural earphone 230 so as to provide the user with a virtual auditory environment of wave sound. Moreover, in the beach scene of a present Example, the number of sound sources is only one, ie, it is the sea.

  When the object shown by the user enters the virtual reality scene, the orientation information providing means 220 acquires the orientation in the virtual reality scene of the object shown by the user by acquiring the physical orientation in the user's real environment, and the object Is provided to the apparatus 10. Taking the direction information providing means 220 as the direction detection means as an example, in the beach scene of the present embodiment, the direction detection means detects the physical direction in which the user exists, and thereby the direction and position of the object in the virtual beach scene. And the change of both are obtained, and the direction information and the direction change information are provided to the apparatus 10. The direction detection means may be provided at any suitable position where the physical direction of the user can be detected. For example, the direction detection means is independently placed on the user's head position, or integrated in the binaural earphone 230 or the virtual reality glasses 240. be able to.

  As an example, assuming that the object's left ear faces the sound source (ie, the sea) direction, the orientation when the object faces the sound source direction is 0 degrees, and the angle increases when the object rotates clockwise. In this case, the azimuth detected by the azimuth detecting means is 90 degrees. The azimuth information acquisition means (not shown) in the apparatus 10 acquires 90 degree azimuth information from the azimuth detection means and transmits it to the identification means (not shown) in the apparatus 10. Thereafter, the specifying unit specifies a set of HRTF data corresponding to the 90-degree azimuth information for the object, and transmits the specific information to a processing unit (not shown) in the apparatus 10. The processing means processes the sound signal of the sound of the wave received from the sound / video player 210 by the sound input means (not shown) in the apparatus 10 based on the set of HRTF data corresponding to the specified 90-degree azimuth information. Then, a binaural audio signal for output is obtained. Audio output means (not shown) in the device 10 provides binaural audio signals to the user via the binaural earphone 230. Since the sound signal of the wave sound from the audio / video player 210 is processed using the set of HRTF data corresponding to the 90 degree azimuth information, the wave sound felt by the user corresponds to the 90 degree azimuth information. The sound of the wave that represents the azimuth feature, that is, felt by the user, is located on the left ear side.

  Furthermore, when the physical orientation of the user in the real environment changes, for example, when the user rotates 90 degrees clockwise, the object in which the user is projected also rotates 90 degrees clockwise in the beach scene, and this is detected by the direction detection means. The orientation change thus made is that the orientation has been rotated 90 degrees clockwise, and the orientation change information is provided to orientation information acquisition means (not shown) in the apparatus 10. After the azimuth information acquisition means acquires the azimuth change information indicating that it has rotated 90 degrees clockwise, the original azimuth information of the object (that is, azimuth information of 90 degrees) and the azimuth change information acquired this time (that is, 90 deg. Clockwise). Based on the direction change information (rotated degree of rotation), the changed direction information (ie, 180 degree direction information) is calculated, and the changed direction information is transmitted to a specifying means (not shown) in the apparatus 10. The specifying means specifies a set of HRTF data corresponding to the changed azimuth information for the object based on the changed azimuth information, and provides the specific information to a processing means (not shown) in the apparatus 10. The processing means processes a binaural audio signal for processing and outputting the sound signal of the wave sound received from the audio / video player 210 by the audio input means (not shown) in the apparatus 10 based on the specified set of HRTF data. Get. Audio output means (not shown) in the device 10 provides binaural audio signals to the user via the binaural earphone 230. Since the sound signal of the wave sound from the audio / video player 210 is processed using a set of HRTF data corresponding to the changed azimuth information (180 degree azimuth information), the wave sound felt by the user is 180. The azimuth | direction characteristic corresponding to the azimuth | direction azimuth | direction information is represented, ie, the sound of the wave felt by the user is located in the back of a user.

  The above-described embodiment is an example in which the direction of the object in the virtual reality scene is rotated 90 degrees clockwise and is not related to the position change of the object. However, those skilled in the art will recognize that the technical solution of the present invention is actually more complicated. It should be explained that it should be understood that it can be applied to object orientation changes. For example, the azimuth change of the object may include parallel movement or a combination of rotation and parallel movement. The translation and rotation are not limited to a two-dimensional plane, and may be any translation and rotation in a three-dimensional space.

  Further, the description of the combination of the virtual auditory environment and the virtual visual environment in the above-described beach application scene is only for better explaining the technical solution of the present invention, and the technical solution of the present invention is applied to the virtual auditory environment. It is necessary to explain that it can be used only for scenes.

  FIG. 3 illustrates a method for controlling audio signal output in a single object virtual auditory environment according to an embodiment of the present invention. The method includes a first acquisition step S310, a first identification step S320, and a first processing step S330.

  With reference to FIG. 3, in the first acquisition step S310, the orientation information of the object in the single object virtual auditory environment is acquired. The orientation information includes the orientation of the object in the single object virtual auditory environment.

  In the first specifying step S320, a set of one or more sound source direction data corresponding to the direction information is specified for the object. The set of one or more sound source azimuth data represents transmission characteristics in which one or more sound sources in a single object virtual auditory environment transmit sound waves in the direction in which the object exists.

  Specifying a set of one or more sound source azimuth data corresponding to the azimuth information for an object can be realized by various methods. For example, based on the azimuth information of the object, one or a plurality of sound source azimuth data matching the azimuth information is matched with the object according to the correspondence relationship between the azimuth information stored in advance and the set of the sound source azimuth data. A set can be selected. For example, based on the azimuth information of the object, a set of one or a plurality of sound source azimuth data corresponding to the azimuth information can be calculated for the object.

  A set of sound source azimuth data is transmitted in a certain direction, such as a set of transfer function data related to the head, a set of time difference data between both ears, and a set of intensity difference data between both ears. It can be any suitable collection of data that can represent the transmission characteristics to be performed.

  In a first processing step S330, one or more sound sources in a single object virtual auditory environment are based on the identified set of one or more sound source orientation data to obtain a binaural audio signal for output. One or more corresponding audio signals are processed respectively.

  In view of the possibility that the orientation of the object in the single object virtual auditory environment may change, in a further embodiment, the changed orientation for the object based on the changed orientation information of the object. By specifying another or a plurality of HRTF data sets corresponding to the information, it is possible to realize an effect of selecting, in real time, a set of HRTF data that matches the current orientation for the object. .

  In this further embodiment, in addition to the first acquisition step, the first specification step, and the first processing step described above, a second acquisition step, a second specification step, and a second processing step are further included.

  In the second acquisition step, the orientation information after the change of the object in the single object virtual auditory environment is acquired. The changed orientation information includes the changed orientation of the object in the single-object virtual auditory environment.

  In the second specifying step, another one or another set of sound source direction data corresponding to the changed direction information is specified for the object. The other one or another set of sound source azimuth data represents a transmission characteristic in which one or more sound sources in the single object virtual auditory environment transmit sound waves to the changed azimuth in which the object exists. .

  Specifying another one or a plurality of sets of sound source direction data corresponding to the changed direction information for the object can be realized by various methods. For example, based on the orientation information after the change of the object, another one that matches the orientation information after the change to the object according to the correspondence relationship between the orientation information stored in advance and the set of sound source orientation data. One or another set of sound source orientation data can be selected. Further, for example, based on the azimuth information after the change of the object, one or another set of a plurality of sound source azimuth data corresponding to the azimuth information after the change can be calculated for the object.

  In a second processing step, one or more in a single object virtual auditory environment based on another identified one or another set of source orientation data to obtain a binaural audio signal for output One or a plurality of audio signals corresponding to each sound source are processed.

Multi-Object Virtual Auditory Environment FIG. 4 shows a device 40 (hereinafter abbreviated as “device 40”) that controls audio signal output in a multi-object virtual auditory environment according to an embodiment of the present invention. In the object virtual auditory environment, there are a plurality of objects. As shown in FIG. 4, the apparatus 40 includes a voice input unit 410, a direction information acquisition unit 420, a specifying unit 430, a processing unit 440, and a voice output unit 450.

  Referring to FIG. 4, voice input means 410 is for receiving a voice signal and transmitting it to processing means 440. For multi-object virtual auditory environments, the audio signal may be, for example, an audio signal collected by a microphone or an audio signal output from an audio player. The audio signal collected by a typical microphone may be an audio signal from each object, and the audio signal output from a typical audio player may be an audio signal from a non-object sound source in a virtual auditory environment. Good. The actual meaning of the audio signal from each object described here is an audio signal that each user collects with a microphone and transmits to the audio input means 410. Here, the audio signal is abbreviated as an audio signal from each object. . Similar to this, the actual meaning of the audio signal from the non-object sound source in the virtual auditory environment described here is the audio signal that the audio player reproduces the audio file and transmits to the audio input means 410, Here, it is abbreviated as an audio signal from a non-object sound source in a virtual auditory environment. The audio signal may be, for example, a monaural, 2-channel or stereo audio signal.

  In order to specify the orientation of the object in the virtual auditory environment, the orientation information acquisition unit 420 acquires the orientation information of each object in the multi-object virtual auditory environment.

In one example, each object receives only audio signals from non-object sound sources in the virtual auditory environment. In this case, the orientation information of each object may include only the orientation of the object in the virtual auditory environment.
In another example, each object receives audio signals from non-object sound sources in a virtual auditory environment and audio signals from other objects, or receives only audio signals from other objects. In these two types, the orientation information of each object includes the orientation of the object in the virtual auditory environment and the relative orientation of the object with respect to other objects.

  The azimuth information of the object can be provided to the azimuth information obtaining unit 420 by the azimuth information providing unit (not shown). The function of the direction information providing means can be realized by various methods, and these methods may be used alone or in combination. For example, each user can be equipped with an orientation detection unit, and the orientation detection unit can detect the orientation of an object reflected in the virtual auditory environment and provide the detected orientation information to the orientation information acquisition unit 420. The orientation detection means may be any suitable element and combination of elements used to detect object orientation, such as a gyro, compass, accelerometer or the like. Alternatively, each user can be equipped with a user input interface, and the user can self-define the orientation of the object projected in the virtual auditory environment and the orientation of the object relative to other objects. The input of the user input interface may include various methods such as clicking, dragging on characters, symbols, sounds, or icons in the interface. The azimuth information input by the user input interface is immediately provided to the azimuth information acquisition unit 420. The detection methods and user self-defining methods described above may be used alone or in combination.

  The orientation information acquisition unit 420 acquires the orientation information of each object and then transmits the orientation information to the identification unit 430. The identification unit 430 corresponds to the orientation information for each object based on the orientation information of each object. A set of a plurality of HRTF data is specified, and the specified information is transmitted to the processing means 440. The specific information may be in various formats, for example, the specific information may include the set of one or more HRTF data itself, or only the index of the specified set of one or more HRTF data. May be included. For the latter, the processing means 440 needs to store a set of HRTF data corresponding to each index.

  The set of one or more HRTF data specified for each object described above represents transmission characteristics in which one or more sound sources in a multi-object virtual auditory environment transmit sound waves in the direction in which the object exists. Used for. When the sound source for an object in the multi-object virtual auditory environment is a single sound source, the specifying unit 430 specifies only one HRTF data set corresponding to the single sound source for the object. When the sound source for an object in the multi-object virtual auditory environment is a plurality of sound sources, the specifying unit 430 corresponds to the plurality of sound sources, and the plurality of HRTF data corresponding to the plurality of sound sources is associated with the object. Identify the set.

  Specifying one or a plurality of sets of HRTF data corresponding to the orientation information for each object by the specifying unit 430 can be realized by various methods. For example, the specifying unit 430 may match one or a plurality of matching information for each object based on the orientation information of each object according to the correspondence between the orientation information stored in advance and the set of HRTF data. Identifies a set of HRTF data. The method of acquiring the correspondence relationship between the orientation information and the set of HRTF data has already been described in the previous sentence, and will not be repeated here for the sake of brevity. Further, for example, based on the orientation information of each object and other related information necessary for calculating the pre-stored HRTF, the identifying unit 430 may provide one or more corresponding to the orientation information for each object. A set of HRTF data can be calculated in real time.

  The processing means 440 is one in the multi-object virtual auditory environment from the voice input means 410 based on a set of one or more HRTF data corresponding to the orientation information specified by the specifying means 430 for each object. Alternatively, one or a plurality of audio signals corresponding to a plurality of sound sources are processed to obtain a binaural audio signal for output. Thereafter, the audio output unit 450 provides binaural audio signals to the corresponding users.

  In view of the possibility that the orientation of each object in the multi-object virtual auditory environment and / or the orientation relative to other objects may change, in a further embodiment, the device 40 may change the orientation of each object after the change. Based on the information, for each object, by specifying another or a plurality of sets of HRTF data corresponding to the changed orientation information, the object is matched with the changed orientation. The effect of selecting a set of HRTF data in real time can also be realized. In the following, this further embodiment will be described in detail.

  When an orientation of an object in a plurality of objects in the virtual auditory environment and / or an orientation relative to another object is changed, orientation information providing means (not shown) installed in the object changes the orientation of the object and / or Alternatively, the azimuth change information can be provided to the azimuth information obtaining unit 420 while obtaining the azimuth change relative to another object.

  In one example, each object receives only audio signals from non-object sound sources in the virtual auditory environment. In this case, the direction change information of each object may include only the direction change in the virtual auditory environment of the object.

  In another example, each object receives audio signals from non-object sound sources in a virtual auditory environment and audio signals from other objects, or receives only audio signals from other objects. In these two types, the orientation information of each object includes a change in orientation of the object in the virtual auditory environment and a change in orientation relative to the other object.

  The function of the direction information providing means can be realized by various methods, and these methods may be used alone or in combination. For example, each user is equipped with an orientation detection means, and the orientation detection means detects the orientation change of the object reflected in the virtual auditory environment in real time, periodically, or in response to a trigger, and detects the orientation change. Information can be provided to the orientation information acquisition means 420. The orientation change of an object in its multi-object virtual auditory environment may include horizontal rotation or translation or a combination thereof and / or vertical rotation or translation or a combination thereof. Alternatively, each user can be equipped with a user input interface, and the user can self-define the orientation change of the object projected in the virtual auditory environment and the orientation change relative to other objects of the object. The input of the user input interface may include various methods such as clicking, dragging on characters, symbols, sounds, or icons in the interface. The direction change information input by the user input interface is immediately provided to the direction information acquisition unit 420. The detection methods and user self-defining methods described above may be used alone or in combination.

  The detection method and the user self-definition method described above can be combined to provide the azimuth information acquisition means 420 with the azimuth change information in the multi-object virtual auditory environment of the object. For example, upon entering the virtual auditory environment, the user may self-define and input the orientation of the projected object in the virtual auditory environment and / or the relative orientation of the object with respect to other objects via the user input interface. The azimuth information can be provided to the azimuth information acquisition means 420 as initial azimuth information. After that, the direction detecting means equipped to each user detects the direction change of the object reflected in the virtual auditory environment in real time, periodically or in response to a trigger, and the detected direction change information is direction information. Provide to the acquisition means 420. In this process, the user can also self-define and adjust the orientation of the object in the virtual auditory environment and / or the relative orientation of the object relative to other objects from time to time through the user input interface.

  The azimuth information acquisition means 420 calculates the azimuth information after the change based on the original azimuth information of each object and the azimuth change information acquired this time after acquiring the azimuth change information of each object in the virtual auditory environment. Is provided to the specifying means 130. The specifying unit 430 specifies one or another set of HRTF data corresponding to the changed direction information for each object based on the changed direction information of each object.

  Alternatively, the orientation information providing means equipped on the object directly acquires the orientation after the object change and / or the relative orientation after the change of another object, and acquires the orientation information after the change. Means 120 may also be provided. In this case, the azimuth information acquisition unit 420 does not need to perform the above-described calculation, and only needs to transmit the acquired azimuth information of each object to the identification unit 430. Based on the azimuth information after the change of the object, another one or another set of HRTF data corresponding to the azimuth information after the change is specified for each object.

  After the specifying unit 430 specifies another set of one or another HRTF data corresponding to the changed orientation information for each object, the processing unit 440 sets the one or more HRTF data. Based on the set, one or more audio signals corresponding to one or more sound sources in the virtual auditory environment from the audio input means 110 are processed to obtain a binaural audio signal for output. Thereafter, the audio output means 450 provides binaural audio signals to the corresponding users.

  An apparatus for controlling audio signal output in a multi-object virtual auditory environment according to an embodiment of the present invention is located on the far end side and can be integrated in a remote server, for example. It is located on the local side and can be integrated into, for example, a local audio / video playback device or multi-party conference device.

  The above-described embodiment acquires direction information of each object in a plurality of objects in an apparatus for controlling audio signal output in a multi-object virtual auditory environment, and one or more corresponding to the direction information for each object. Although an example of identifying a set of HRTF data has been described as an example, those skilled in the art will understand that, in another embodiment, an apparatus that controls the output of the audio signal only includes orientation information of some objects in a plurality of objects. It is necessary to explain that it should be understood that, for some of the objects, one or more sets of HRTF data corresponding to the orientation information can be specified for each of the objects. For example, if the device that controls the output of the audio signal is located on the local side, only the azimuth information of the local object is acquired, and for the local object, one or more HRTF data corresponding to the azimuth information The local object orientation information may include a local object orientation in a multi-object virtual auditory environment and / or a relative orientation relative to other objects. The number of local objects may be one or more.

  In the latter part, the multi-object virtual auditory environment for simulating a multi-party conference scene is taken as an example, and FIG. 5 is used as an example to explain in more detail the utilization of the apparatus for controlling the audio signal output of FIG. .

  FIG. 5 shows an exemplary multi-object virtual reality system 50 according to an embodiment of the present invention. As shown in FIG. 5, the multi-object virtual reality system 50 includes a multi-party conference device 510, a device 40 (hereinafter abbreviated as “device 40”) for controlling the audio signal output of FIG. 4, a user input interface 520, binaural Including earphones 530 and virtual reality glasses 540. In this application scene, the case where the apparatus 40 is located on the local side and receives only the orientation information of the local object and the changed orientation information will be described as an example.

  Referring to FIG. 5, a multi-party conference device 510 provides a video signal and an audio signal from a participant, and the video signal is for simulating a virtual visual environment of a multi-party conference scene, The audio signal is for simulating a virtual auditory environment in a multi-party conference scene. When the user wears the binaural earphone 530 and the virtual reality glasses 540, the multi-party conference device 510 transmits a video signal to the virtual reality glasses 540 so as to present a virtual multi-party conference scene to the user. In the virtual multi-party conference scene, the user and other participants can be presented as objects located in different directions. Based on this, the multi-party conference equipment 510 further transmits audio signals from other participants processed by the device 40 to the binaural earphone 530 so as to simulate the audio of the object located in different directions. .

  In the multi-party conference scene of this embodiment, there are a total of three participants, each of which is a local user A located at the location of the device 40 and a non-local participant B and C located at different locations. I will do it. The local user A is wearing the above-described binaural earphone 530 and virtual reality glasses 540. After the multi-party conferencing equipment 510 is activated, the local user A is shown on the local object a in the virtual reality scene, and the non-local participants B and C are shown on the other objects b and c in the virtual reality scene, respectively. Is done. At this time, the local user A can see the objects b and c in the virtual reality scene through the virtual reality glasses 540, and can also see the object a reflected in the virtual reality scene, or at least a part thereof.

  In this embodiment, the local user A uses the user input interface 520 to determine the orientation of the local object a in which the user is reflected in the multi-party conference scene and the relative orientation of the object a relative to the other objects b and c. While defining and inputting, this azimuth | direction information can be provided to the azimuth | direction information acquisition means (not shown) in the apparatus 40 as initial azimuth | direction information.

  The orientation information acquisition means (not shown) in the device 40 acquires the orientation information and transmits it to the specifying means (not shown) in the device 40. The specifying means corresponds to the orientation information for the local object a 2. One set of HRTF data is specified, and the specified information is transmitted to processing means (not shown) in the apparatus 40. One set of HRTF data represents transmission characteristics in which sound waves emitted from the object b as a sound source are transmitted to the object a, and the other set of HRTF data transmits sound waves emitted from the object c as a sound source to the object a. Represents the transmission characteristics to be performed. Thereafter, the processing means (not shown) in the device 40 is based on the objects b and c received from the multi-party conference device 510 by the voice input means (not shown) in the device 40 based on the set of two specified HRTF data. The binaural audio signal for processing and outputting is obtained. Audio output means (not shown) in device 40 provides binaural audio signals to local user A via binaural earphones 530. Since the audio signals from the objects b and c are processed by the processing means in the device 40 based on different sets of HRTF data corresponding to the direction information, the audio from the objects b and c felt by the local user A is The azimuth | direction characteristic corresponding to azimuth | direction information is represented. In accordance with such a plan, the user can identify the orientations of different participants based only on audio in a multi-party conference of a virtual reality scene. For example, assuming that local user A is positioned in the middle, local object a is in the middle, object b is in the front left, and object c is in the right front, The voice of the participant B that A actually heard in the virtual multi-party conference is in the left front, the voice of the participant C is in the right front, and the user A has set or the user A Since there is no portion that does not match the virtual visual scene seen through the virtual reality glasses 540, the user's virtual reality experience is enhanced in hearing.

  Further, the local user A can adjust the orientation of the local object a in the multi-party conference scene and the relative orientation of the object a with respect to the other objects b and c by the user input interface 520 at any time.

  Orientation information acquisition means (not shown) in the apparatus 40 acquires the changed orientation information and transmits it to the specifying means (not shown) in the apparatus 40, and the specifying means changes the orientation information for the local object a. Two sets of HRTF data corresponding to are identified. One set of HRTF data represents transmission characteristics in which sound waves emitted from the object b as a sound source are transmitted to the object a, and the other set of HRTF data transmits sound waves emitted from the object c as a sound source to the object a. Represents the transmission characteristics to be performed. Thereafter, the processing means (not shown) in the device 40 is based on the objects b and c received from the multi-party conference device 510 by the voice input means (not shown) in the device 40 based on the set of two specified HRTF data. The binaural audio signal for processing and outputting is obtained. Audio output means (not shown) in device 40 provides binaural audio signals to local users via binaural earphones 530. Since the audio signals from the objects b and c are processed by the processing means in the device 40 based on the set of HRTF data corresponding to the changed azimuth information, the audio signals from the objects b and c felt by the local user A are obtained. The voice represents a change corresponding to the change of the direction information. According to such a plan, the user can identify a change in the direction of different participants based only on the sound in the multi-party conference of the virtual reality scene. For example, based on the azimuth information that the user A has previously defined and entered, the azimuth information after the change that the user A has self-defined and entered is such that the local object a is not changed and the object b is located in the middle. Assuming that object C has moved to the right and object c has moved to its right, participant B's voice that user A actually heard in the virtual multi-party conference moved to the front, The voice of the party C moves to the right side, and since there is no part that does not match the virtual visual scene set by the user A or seen through the virtual reality glasses 540, the user's virtual reality experience is further enhanced in hearing. doing.

  It can be understood that the two non-local participants B and C can also be provided with a device 40 on each local side to provide a similar virtual auditory environment.

  The combination of the virtual auditory environment and the virtual visual environment in the above-described multi-party conference application scene is only described for better explaining the technical solution of the present invention, and the technical solution of the present invention is applied to the virtual auditory environment. It can be understood that it can be used only for scenes.

  FIG. 6 illustrates a method for controlling audio signal output in a multi-object virtual auditory environment according to an embodiment of the present invention. The method includes a first acquisition step S610, a first identification step S620, and a first processing step S630.

  Referring to FIG. 6, in the first acquisition step S610, the orientation information of at least one object among a plurality of objects in the multi-object virtual auditory environment is acquired. The orientation information of the at least one object includes the orientation of the object in a multi-object virtual auditory environment and / or a relative orientation with respect to other objects.

  In the first specifying step S620, a set of one or more sound source direction data corresponding to the direction information is specified for the at least one object. The set of one or more sound source direction data represents transmission characteristics in which one or more sound sources in the multi-object virtual auditory environment transmit sound waves in the direction in which the at least one object exists.

  Specifying a set of one or more sound source direction data corresponding to the direction information for at least one object can be realized by various methods. For example, based on the azimuth information of the at least one object, one or more matching the azimuth information to the at least one object according to the correspondence between the azimuth information stored in advance and the set of sound source azimuth data, or A set of a plurality of sound source direction data can be selected. Also, for example, based on the orientation information of the at least one object, a set of one or more sound source orientation data corresponding to the orientation information can be calculated for the at least one object.

  A set of sound source azimuth data is transmitted in a certain direction, such as a set of transfer function data related to the head, a set of time difference data between both ears, and a set of intensity difference data between both ears. It can be any suitable collection of data that can represent the transmission characteristics to be performed.

  Corresponding to one or more sound sources in a multi-object virtual auditory environment based on the identified set of one or more sound source orientation data to obtain a binaural audio signal for output in a first processing step S630 One or more audio signals to be processed.

  In one example, for each object in the at least one object, the other object is one or more sound sources in the multi-object virtual auditory environment.

  In view of the possibility that the orientation of the at least one object in the multi-object virtual auditory environment may change, in a further embodiment, the at least one object is based on the changed orientation information of the at least one object. By specifying another one or another set of HRTF data corresponding to the changed orientation information for the object, the set of HRTF data that matches the current orientation is assigned to the at least one object. The effect of selecting in real time can also be realized.

  In this further embodiment, in addition to the first acquisition step, the first specification step, and the first processing step described above, a second acquisition step, a second specification step, and a second processing step are further included.

  In the second acquisition step, the orientation information after the change of the at least one object is acquired. The changed orientation information of the at least one object includes a changed orientation of the object in the virtual auditory environment and / or a changed relative orientation with respect to another object.

  In the second specifying step, another one or another set of sound source direction data corresponding to the changed direction information is specified for the at least one object. The other one or another set of sound source azimuth data is a transmission characteristic in which one or more sound sources in a multi-object virtual auditory environment transmit sound waves to a changed azimuth in which the at least one object exists. Represent each.

  It is also possible to specify another one or another set of sound source azimuth data corresponding to the changed azimuth information for the at least one object by various methods. For example, based on the azimuth information after the change of the at least one object, according to the correspondence between the azimuth information stored in advance and the set of sound source azimuth data, the azimuth information after the change is assigned to the at least one object. One or another set of sound source azimuth data that matches can be selected. Also, for example, based on the azimuth information after the change of the at least one object, another one or another set of sound source azimuth data corresponding to the azimuth information after the change is calculated for the at least one object. can do.

  In a second processing step, one or more in the multi-object virtual auditory environment is based on another identified one or another set of sound source orientation data to obtain a binaural audio signal for output. One or more audio signals corresponding to the sound source are processed.

  Each means in the apparatus for controlling the audio signal output according to each embodiment of the present invention can be realized completely or partly by software, hardware, firmware, or any combination thereof. Examples of hardware means include, but are not limited to, system on chip (SOC), integrated circuit (IC), application specific integrated circuit (ASIC), field programmable gate array (FPGA), and the like.

The apparatus for controlling the output of the audio signal according to each embodiment of the present invention may be realized as a single device, that is, all means may be realized in one physical device. Alternatively, the apparatus for controlling the audio signal output according to each embodiment of the present invention may be a distributed device, that is, a part of means and another means are respectively arranged in different physical devices or positions. .
Between the device for controlling the audio signal output according to each embodiment of the present invention and an external device, or between the means within the device for controlling the audio signal output according to each embodiment of the present invention, any currently known, Alternatively, they may communicate with each other by future-developed communication technologies, including but not limited to process / thread communication, Bluetooth, infrared, wireless or wired computer networks, communication networks, and the like.

  The above description of the present invention is intended to enable any person skilled in the art to make or use the invention. Various modifications of the present invention will be obvious to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit and scope of protection of the present disclosure. . Thus, the present invention is not limited to the examples and designs of this paper, but is consistent with the widest range of principles and novelty characteristics disclosed in this paper.

10 apparatus 110 voice input means 120 orientation information acquisition means 130 identification means 140 processing means 150 voice output means

Claims (27)

A method of controlling audio signal output in a virtual auditory environment,
A first acquisition step of acquiring orientation information of the object in the virtual auditory environment;
One or more sound source azimuth data corresponding to the azimuth information, each representing transmission characteristics of one or more sound sources in the virtual auditory environment for transmitting the sound waves in the azimuth corresponding to the azimuth information. A first identification step for identifying a set of
First processing each of one or more audio signals corresponding to the one or more sound sources based on the set of one or more sound source orientation data to obtain a binaural audio signal for output. Processing steps;
Including a method. A second acquisition step of acquiring orientation information after the change of the object in the virtual auditory environment;
One or more sound sources in the virtual auditory environment for the object, each representing a transmission characteristic for transmitting a sound wave in a direction corresponding to the changed direction information; another corresponding to the changed direction information A second specifying step of specifying one or another set of sound source direction data;
In order to obtain a binaural audio signal for output, one or more audio signals corresponding to the one or more sound sources are respectively obtained based on the other one or another set of sound source direction data. A second processing step to process;
The method of claim 1, further comprising:   In the first specifying step, based on the azimuth information, the object is matched with the azimuth information according to a correspondence relationship between azimuth information stored in advance and a set of sound source azimuth data. The method of claim 1, comprising selecting a set of sound source orientation data.   The method according to claim 1, wherein the first specifying step includes calculating a set of the one or more sound source azimuth data corresponding to the azimuth information for the object based on the azimuth information.   The second specifying step matches the object with the changed azimuth information according to a correspondence relationship between the azimuth information stored in advance and a set of sound source azimuth data based on the changed azimuth information. 3. The method of claim 2, comprising selecting the one or more other sets of sound source orientation data.   The second specifying step calculates, based on the changed azimuth information, the one or another set of sound source azimuth data corresponding to the changed azimuth information for the object. The method of claim 2 comprising. The object is one object;
The first obtaining step includes obtaining azimuth information of the one object including an azimuth in the virtual auditory environment of the one object;
The first specifying step corresponds to the azimuth information for the one object, each representing a transmission characteristic in which one or more sound sources in the virtual auditory environment transmit sound waves to the azimuth of the one object. The method of claim 1, comprising identifying a set of one or more sound source orientation data to perform. The object is one object;
The second obtaining step includes obtaining azimuth information after the change of the one object, including the azimuth after the change in the virtual auditory environment of the one object,
The second specifying step represents, for the one object, a transmission characteristic in which one or a plurality of sound sources in the virtual auditory environment transmit sound waves in the changed direction of the one object, respectively. 3. The method of claim 2, comprising identifying another one or more sets of sound source orientation data corresponding to subsequent orientation information. The object is a plurality of objects;
The first obtaining step includes obtaining orientation information of at least one object in the plurality of objects, including orientation of the object in the virtual auditory environment and / or orientation relative to other objects;
The first specifying step represents, for each of the at least one object, one or more sound sources in the virtual auditory environment that indicate transmission characteristics for transmitting sound waves in the direction of the at least one object, respectively. The method of claim 1, comprising identifying a set of one or more sound source orientation data corresponding to. The object is a plurality of objects;
The second obtaining step includes a changed orientation of the object in the virtual auditory environment and / or a changed relative orientation with respect to another object, and a changed orientation of at least one object in the plurality of objects. Including obtaining information,
The second specifying step represents, for the at least one object, transmission characteristics in which one or more sound sources in the virtual auditory environment transmit sound waves to the changed orientation of the at least one object, respectively. The method according to claim 2, further comprising identifying another one or another set of sound source orientation data corresponding to the changed orientation information.   The method according to claim 9 or 10, wherein for each object in the at least one object, another object is the one or more sound sources in the virtual auditory environment. The set of sound source direction data includes
A set of transfer function data related to the head,
A set of time difference data between both ears;
The method according to any one of claims 1 to 10, wherein any one of a set of intensity difference data between both ears is included. A device for controlling audio signal output in a virtual auditory environment,
Audio input means arranged to receive one or more audio signals corresponding to one or more sound sources in the virtual auditory environment;
Azimuth information acquisition means arranged to acquire azimuth information of objects in the virtual auditory environment;
One or more sound source azimuth data corresponding to the azimuth information, each representing transmission characteristics of one or more sound sources in the virtual auditory environment for transmitting the sound waves in the azimuth corresponding to the azimuth information. Identifying means arranged to identify a set of
Processing means arranged to process each of the one or more audio signals based on the set of one or more sound source orientation data to obtain a binaural audio signal;
Audio output means arranged to output the binaural audio signal;
An apparatus comprising: The azimuth information acquisition means is further arranged to acquire azimuth information after the change of the object in the virtual auditory environment,
The specifying means further represents the changed azimuth for each of the objects, each representing a transmission characteristic in which one or a plurality of sound sources in the virtual auditory environment transmit sound waves in a direction corresponding to the changed azimuth information. Arranged to identify another one or another set of sound source orientation data corresponding to the information,
The processing means is further arranged to process each of the one or more audio signals based on the other one or another set of sound source azimuth data to obtain a binaural audio signal. The apparatus of claim 13.   The azimuth information acquisition means is arranged to receive the azimuth change information of the object in the virtual auditory environment and to specify the azimuth information after the change of the object based on the azimuth change information and the azimuth information. The apparatus of claim 14.   The specifying unit is configured to match the object with the one or more sound source directions according to the correspondence relationship between the stored direction information and the set of sound source direction data based on the direction information. The apparatus of claim 13, arranged to select a collection of data.   The apparatus according to claim 13, wherein the specifying unit is arranged to calculate the set of the one or more sound source azimuth data corresponding to the azimuth information based on the azimuth information.   The specifying means matches the object with the changed azimuth information according to the correspondence relationship between the azimuth information stored in advance and the set of sound source azimuth data based on the changed azimuth information. 15. The apparatus of claim 14, arranged to select one or another set of sound source orientation data.   The specifying means is arranged to calculate, on the object, the other one or another set of a plurality of sound source direction data corresponding to the changed direction information based on the changed direction information. The apparatus according to claim 14. The object is one object;
The azimuth information acquisition means is arranged to acquire azimuth information of the one object including the azimuth in the virtual auditory environment of the one object,
The specifying unit corresponds to direction information that represents transmission characteristics of one or a plurality of sound sources in the virtual auditory environment for transmitting the sound wave to the direction of the one object for the one object, respectively. 14. The apparatus of claim 13, arranged to identify a set of one or more sound source orientation data. The object is one object;
The azimuth information acquisition means is arranged to acquire the azimuth information after the change of the one object, including the azimuth after the change in the virtual auditory environment of the one object,
The specifying means represents, for the one object, a transmission characteristic in which one or a plurality of sound sources in the virtual auditory environment transmit sound waves to the changed direction of the one object, respectively, The apparatus according to claim 14, arranged to identify another one or another set of sound source orientation data corresponding to orientation information. The object is a plurality of objects;
The azimuth information acquisition means is arranged to acquire azimuth information of at least one object in the plurality of objects including an azimuth of the object in the virtual auditory environment and / or a relative azimuth relative to another object,
The specifying means corresponds to the orientation information, each of which represents a transmission characteristic in which one or more sound sources in the virtual auditory environment transmit sound waves to the orientation of the at least one object with respect to the at least one object. 14. The apparatus of claim 13, arranged to identify a set of one or more sound source orientation data to be The object is a plurality of objects;
The azimuth information acquisition means includes a changed azimuth of the object in the virtual auditory environment and / or a changed azimuth relative to another object, and the changed azimuth of at least one object in the plurality of objects. Arranged to get information,
The specifying means represents, for the at least one object, a transmission characteristic in which one or more sound sources in the virtual auditory environment transmit sound waves in the changed orientation of the at least one object, respectively. 15. The apparatus of claim 14, arranged to identify another one or another set of sound source orientation data corresponding to subsequent orientation information.   24. The apparatus of claim 22 or 23, wherein for each object in the at least one object, another object is the one or more sound sources in the virtual auditory environment.   The apparatus according to claim 13, further comprising orientation information providing means arranged to provide the orientation information acquisition means with the orientation information of the object.   15. The apparatus according to claim 14, further comprising orientation information providing means arranged to provide the orientation information acquisition means with the changed orientation information of the object or the orientation change information of the object. The set of sound source direction data includes
A set of transfer function data related to the head,
A set of time difference data between both ears;
The apparatus according to any one of claims 13 to 23, wherein any one of a set of intensity difference data between both ears is included.