JP5983421B2 - Audio processing apparatus, audio processing method, and audio processing program - Google Patents

Description

  The present invention relates to a voice processing device, a voice processing method, and a voice processing program.

  There is a technique for localizing a sound image of a virtual sound source at an arbitrary position around a listener using a playback device that outputs left and right channels such as headphones and earphones. In this technique, sound image localization is realized by performing a convolution operation on a head-related transfer function (HRTF) corresponding to the position of the virtual sound source in an audio signal output from the virtual sound source. It is also possible to arrange a plurality of virtual sound sources around the listener and localize the sound image of each virtual sound source.

  Moreover, there are the following technologies related to the virtual sound source. For example, when playing a performance or sound, a virtual sound source is formed by wavefront synthesis of sound waves output from a speaker array in order to realize natural and lively and realistic reproduction, and the position of the virtual sound source There has been proposed a technique for changing the value in the vicinity thereof.

  In addition, a technology has been proposed that obtains a high wrapping feeling that emphasizes the localization of the virtual sound image according to the position of the center of gravity by multiplying the signal that creates the virtual sound image by the weighting coefficient calculated from the center of gravity of the input signal of each channel. ing.

JP 2006-86921 A JP2011-21112A

  In the process of localizing the sound images of a plurality of virtual sound sources, a convolution operation using a head-related transfer function corresponding to the position of each virtual sound source is executed by the number of virtual sound sources. For this reason, there is a problem that the processing load increases as the number of virtual sound sources increases. In this regard, the audio signal corresponding to each of the plurality of virtual sound sources is distributed to a certain number of virtual speakers, and the convolution calculation is performed using the HRTF corresponding to each virtual speaker, so that the processing amount of the convolution calculation is always reduced to the virtual speaker. The number can be fixed. However, in the method using the virtual speaker as described above, since the HRTF corresponding to the position of the virtual speaker different from the position of the virtual sound source is used, the localization feeling of the sound image of the virtual sound source may be ambiguous.

  An object of one aspect is to provide an audio processing device, an audio processing method, and an audio processing program that can improve the sense of localization of sound images of a plurality of virtual sound sources by low-load processing.

  In one scheme, a speech processing device is provided. The speech processing apparatus has a speaker arrangement unit and a speech synthesis unit. A speaker arrangement | positioning part calculates the angle of the direction along the circumference centering on a listener between two virtual sound sources adjacent to the direction along the circumference seeing from a listener. The speaker placement unit is sandwiched between virtual sound sources whose calculated angles are equal to or greater than a threshold value in a range in a direction along the circumference with the listener as the center, and each virtual sound source viewed from the listener A plurality of virtual speakers less than the number of the plurality of virtual sound sources arranged around the listener is arranged in the second range excluding the first range that does not include the position in the direction. The speech synthesizer distributes audio signals from each of the plurality of virtual sound sources to one or more virtual speakers selected for each virtual sound source among the plurality of virtual speakers. The speech synthesizer synthesizes the audio signal distributed to each virtual speaker into two left and right channel audio signals using a head-related transfer function corresponding to the position of each virtual speaker.

Further, in one proposal, a voice processing method is provided that executes the same processing as that realized by the voice processing device.
Furthermore, in one proposal, a voice processing program that causes a computer to execute the same processing as that of the voice processing device described above is provided.

  In one aspect, the localization of sound images of a plurality of virtual sound sources can be improved by low-load processing.

It is a figure which shows the example of the audio | voice processing apparatus of 1st Embodiment. It is a figure which shows the example of the speech processing system of 2nd Embodiment. It is a figure which shows the hardware structural example of a speech processing unit. It is a figure which shows the hardware structural example of a user terminal. It is a block diagram which shows the function example of a speech processing unit. It is a figure shown about the example of a production | generation of the audio | voice signal of a channel on either side. It is a figure which shows the example of the positional relationship of a sound source and a virtual speaker. It is a figure which shows the example of the arrangement | positioning method of a virtual speaker when the number of virtual speakers is more than the number of virtual sound sources. It is a 1st figure which shows the example of the arrangement | positioning method of a virtual speaker when the number of virtual speakers is less than the number of virtual sound sources. It is a 2nd figure which shows the example of the arrangement | positioning method of a virtual speaker when the number of virtual speakers is less than the number of virtual sound sources. It is a 3rd figure which shows the example of the arrangement | positioning method of a virtual speaker when the number of virtual speakers is less than the number of virtual sound sources. It is a 4th figure which shows the example of the arrangement | positioning method of a virtual speaker when the number of virtual speakers is less than the number of virtual sound sources. It is a 5th figure which shows the example of the arrangement | positioning method of a virtual speaker when the number of virtual speakers is less than the number of virtual sound sources. It is a figure shown about the example of a user status table. It is a figure shown about the example of a sound source management table. It is a figure shown about the example of a virtual speaker position table. It is a figure shown about the example of arrangement | positioning information. It is a figure shown about the example of an arrangement | positioning area | region management table. It is a flowchart which shows the example of the arrangement | positioning process of a virtual speaker. It is a flowchart (continuation) which shows the example of arrangement | positioning processing of a virtual speaker. It is a 1st figure which shows the example of arrangement | positioning of a virtual speaker. It is a 2nd figure which shows the example of arrangement | positioning of a virtual speaker. It is a 3rd figure which shows the example of arrangement | positioning of a virtual speaker. It is a 4th figure which shows the example of arrangement | positioning of a virtual speaker. It is a 5th figure which shows the example of arrangement | positioning of a virtual speaker. It is a figure which shows the example of the setting method of the weight with respect to an arrangement | positioning area | region. It is a figure which shows the modification of arrangement | positioning of the virtual speaker which considered the weight.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating an example of a speech processing apparatus according to the first embodiment. The audio processing device 1 synthesizes audio signals from a plurality of virtual sound sources arranged around the listener into left and right channel audio signals. The virtual sound source is a sound source that is virtually arranged at an arbitrary position around the listener in order to express sound. Note that the position of each virtual sound source is set in advance by, for example, a user input operation.

  The speech processing apparatus 1 includes a speaker placement unit 2 and a speech synthesis unit 3. The processing of the speaker placement unit 2 and the speech synthesis unit 3 is performed by, for example, a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). It is realized by another electronic circuit or a combination of a processor and another electronic circuit.

  The speaker placement unit 2 dynamically places a plurality of virtual speakers less than the number of virtual sound sources on a predetermined circumference centered on the listener. The virtual speaker means a sound that can be heard from a speaker arrangement position that does not actually exist by sound field processing. Note that the virtual speaker is also a kind of virtual sound source, but in this embodiment, as will be described later, a generation source of an audio signal distributed to each virtual speaker is referred to as a “virtual sound source” and is dynamically generated by the speaker placement unit 2. The virtual sound source arranged in is called “virtual speaker”.

  The voice synthesizer 3 distributes audio signals from each of the plurality of virtual sound sources to one or more virtual speakers selected for each virtual sound source among the plurality of virtual speakers. The voice synthesizer 3 synthesizes a voice signal distributed to each virtual speaker into a left and right two-channel voice signal using an HRTF (head related transfer function) corresponding to the position of each virtual speaker.

  In the voice synthesizer 3, for each virtual speaker, a convolution operation is performed using the voice signal distributed to the virtual speaker and the HRTF corresponding to the position of the virtual speaker. Thereby, the sound image of the sound distributed to the virtual speaker is localized at the position of the virtual speaker. Here, since the number of virtual speakers is smaller than the number of virtual sound sources, the calculation load is reduced compared to the case where the convolution operation is performed using the HRTF corresponding to the position of each virtual sound source. Further, for example, by setting the number of virtual speakers to a constant number smaller than the number of virtual sound sources, the number of convolution calculations can be kept constant regardless of the number of virtual sound sources.

  The speaker placement unit 2 places a plurality of virtual speakers in the following procedure. The speaker placement unit 2 calculates an angle in a direction along the circumference with the listener as a center, between two virtual sound sources adjacent to each other in the direction along the circumference as viewed from the listener. The speaker placement unit 2 is sandwiched between virtual sound sources whose calculated angles are equal to or greater than a threshold value from a range in a direction along the circumference with the listener as the center, and positions in the direction of these virtual sound sources. A first range that does not include is determined. And the speaker arrangement | positioning part 2 arrange | positions said several virtual speaker in the 2nd range except a 1st range among the range of the direction along said circumference centering on a listener, 1st No virtual speaker is placed in the range.

  Thereby, the angle between the straight line connecting the virtual speaker and the listener and the straight line connecting the virtual sound source adjacent to the virtual speaker and the listener with respect to the direction along the circumference can be reduced. As a result, the feeling of localization of the sound image of each virtual sound source can be improved.

Here, an example in which three virtual speakers 6a to 6c are arranged when four virtual sound sources 5a to 5d are arranged around the listener 4 as shown in FIG. 1 will be described.
The speaker placement unit 2 calculates the angles θ1 to θ4 based on the positions of the listener 4 and the virtual sound sources 5a to 5d. Here, it is assumed that the speaker arrangement unit 2 calculates an angle in the clockwise direction along the circumference 7 with the listener 4 as the center. The angle θ1 is an angle between a straight line connecting the listener 4 and the virtual sound source 5a and a straight line connecting the listener 4 and the virtual sound source 5b. The angle θ2 is an angle between a straight line connecting the listener 4 and the virtual sound source 5b and a straight line connecting the listener 4 and the virtual sound source 5c. The angle θ3 is an angle between a straight line connecting the listener 4 and the virtual sound source 5c and a straight line connecting the listener 4 and the virtual sound source 5d. The angle θ4 is an angle between a straight line connecting the listener 4 and the virtual sound source 5d and a straight line connecting the listener 4 and the virtual sound source 5a.

  Here, it is assumed that the angles θ1, θ2, and θ3 are less than a predetermined angle, and the angle θ4 is not less than the predetermined angle. The predetermined angle is, for example, a number obtained by dividing 360 ° by the number of speakers arranged.

  In this case, the speaker placement unit 2 is sandwiched between the virtual sound source 5d and the virtual sound source 5a adjacent to each other in the range of the angle θ4 on the circumference 7, and the direction of the virtual sound sources 5a and 5d as viewed from the listener 4 A range that does not include is determined as the first range described above. This first range does not include the line segments connecting the listener 4 and the virtual sound sources 5a and 5d. The speaker placement unit 2 does not place a virtual speaker in the first range. On the other hand, the speaker arrangement unit 2 arranges the virtual speakers 6a, 6b, and 6c on the circumference 7 in a second range (that is, a range of angles θ1, θ2, and θ3) excluding the first range. This second range includes line segments that connect the listener 4 and the virtual sound sources 5a and 5d, respectively.

  Thereby, the angle between the straight line connecting the virtual speaker and the listener and the straight line connecting the virtual sound source adjacent to the virtual speaker and the listener with respect to the direction along the circumference 7 can be reduced. Here, when the audio signal output from the virtual sound source is distributed to the virtual speaker, the angle between the straight line connecting the virtual speaker and the listener and the straight line connecting the virtual sound source adjacent to the virtual speaker and the listener should be smaller. The feeling of localization of the sound image of the virtual sound source is improved. This is because the error between the HRTF that should be used originally corresponding to the position of the virtual sound source and the HRTF used in the actual calculation corresponding to the position of the virtual speaker is small. Therefore, it is possible to improve the sense of localization of the sound images of a plurality of virtual sound sources.

[Second Embodiment]
FIG. 2 is a diagram illustrating an example of a speech processing system according to the second embodiment.
The voice processing system shown in FIG. 2 includes a voice processing device 100 that performs control processing for providing voice information to a user. A plurality of access points 21 a to 21 d for transmitting and receiving wireless signals are connected to the audio processing device 100 via the network 10. The network 10 is, for example, a LAN (Local Area Network). In this case, the access points 21a to 21d are wireless LAN access points.

On the other hand, the user carries the user terminal 200 and the headphones 12. The user terminal 200 can wirelessly communicate with the access points 21a to 21d.
The voice processing device 100 synthesizes a voice signal output from a sound source virtually arranged by an administrator or the like, and transmits the synthesized voice signal to the user terminal 200 through at least one of the access points 21a to 21d. . Hereinafter, “sound source” refers to a virtual sound source arranged in advance in a virtual space corresponding to the environment around the user.

  In addition, the voice processing device 100 has a function of detecting the position of the user terminal 200. As an example in the present embodiment, the speech processing apparatus 100 receives signals transmitted from the user terminal 200 from the access points 21a to 21d, and detects the position of the user terminal 200 based on these received signals. For example, the speech processing apparatus 100 receives signals transmitted from the user terminal 200 through the access points 21a to 21d, and triangulation based on the difference in signal reception time or the difference in received radio wave intensity at each access point. Is used to detect the position of the user terminal 200. When this method is used, at least three access points used for position detection are installed.

  The headphone 12 is a device that converts an analog audio signal into sound waves using a built-in speaker. The headphone 12 includes a driver unit (not shown) that reproduces and outputs an analog audio signal output from the user terminal 200. In addition, the sensor 11 is mounted on the headphone 12.

  The sensor 11 detects the direction in which the user is facing. Hereinafter, the direction detected by the sensor 11 is referred to as “line-of-sight direction”. The sensor 11 includes, for example, an acceleration sensor, a gyro sensor, and a geomagnetic sensor. The sensor 11 may be provided at a position different from the headphones 12 or may be provided at a position other than the head. However, the purpose of the sensor 11 is to detect where the user is looking. For this reason, it is desirable that the sensor 11 be provided on the user's head. Further, the direction detected by the sensor 11 may be a two-dimensional direction along a horizontal plane or a three-dimensional direction including a vertical direction.

  The user terminal 200 converts the audio signal received from the audio processing device 100 into an analog signal, and outputs the converted analog audio signal to the driver unit of the headphones 12. In addition, the user terminal 200 calculates the user's line-of-sight direction based on the detection result of the sensor 11, and transmits the calculated line-of-sight direction to the voice processing device 100 through at least one of the access points 21a to 21d.

  FIG. 3 is a diagram illustrating a hardware configuration example of the sound processing device. The sound processing apparatus 100 includes a processor 101, a RAM (Random Access Memory) 102, an HDD (Hard Disk Drive) 103, an image signal processing unit 104, an input signal processing unit 105, a disk drive 106, and a communication interface 107. The unit is connected to the bus 108 in the sound processing apparatus 100.

  The processor 101 is a processor including an arithmetic unit that executes program instructions. The processor 101 loads at least a part of the program and data stored in the HDD 103 into the RAM 102 and executes the program. The processor 101 may include a plurality of processor cores. Moreover, the voice processing apparatus 100 may include a plurality of processors. The voice processing apparatus 100 may perform parallel processing using a plurality of processors or a plurality of processor cores. A set of two or more processors, a dedicated circuit such as an FPGA or an ASIC, a set of two or more dedicated circuits, a combination of a processor and a dedicated circuit, and the like may be referred to as a “processor”.

  The RAM 102 is a volatile memory that temporarily stores programs executed by the processor 101 and data referred to by the programs. Note that the audio processing device 100 may include a memory of a type other than the RAM, or may include a plurality of volatile memories.

  The HDD 103 is a non-volatile storage device that stores software programs and data such as an OS (Operating System), firmware, and application software. Note that the voice processing apparatus 100 may include other types of storage devices such as a flash memory, and may include a plurality of nonvolatile storage devices.

  The image signal processing unit 104 outputs an image to the display 13 connected to the sound processing apparatus 100 in accordance with a command from the processor 101. As the display 13, a CRT (Cathode Ray Tube) display, a liquid crystal display, or the like can be used.

  The input signal processing unit 105 acquires an input signal from the input device 14 connected to the sound processing apparatus 100 and notifies the processor 101 of the input signal. As the input device 14, a pointing device such as a mouse or a touch panel, a keyboard, or the like can be used.

  The disk drive 106 is a drive device that reads programs and data recorded on the recording medium 15. As the recording medium 15, for example, a magnetic disk such as a flexible disk (FD) or HDD, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk (MO). Can be used. The disk drive 106 stores the program and data read from the recording medium 15 in the RAM 102 or the HDD 103 in accordance with an instruction from the processor 101.

  The communication interface 107 transmits / receives data to / from other devices (for example, the user terminal 200) through the network 10. The communication interface 107 performs demodulation / decoding of a received signal, encoding / modulation of a transmission signal, and the like.

  Note that the audio processing apparatus 100 may not include the disk drive 106, and may not include the image signal processing unit 104 or the input signal processing unit 105 when accessed exclusively from another information processing apparatus. . Further, the display 13 and the input device 14 may be formed integrally with the casing of the sound processing apparatus 100.

  FIG. 4 is a diagram illustrating a hardware configuration example of the user terminal. The user terminal 200 includes a processor 201, a RAM 202, a flash memory 203, a display 204, an input unit 205, an input interface 206, an output interface 207, and a wireless interface 208. The unit is connected to the bus 209 in the user terminal 200.

  The processor 201 is a processor including an arithmetic unit that executes program instructions, like the processor 101 described above. The RAM 202 is a volatile memory that temporarily stores programs executed by the processor 201 and data, like the RAM 102 described above.

  The flash memory 203 is a non-volatile storage device that stores programs and data such as an OS, firmware, and application software. Note that the user terminal 200 may include another type of storage device such as an HDD, or may include a plurality of nonvolatile storage devices.

  The display 204 displays an image according to a command from the processor 201. As the display 204, a liquid crystal display (LCD), an organic EL (Electro Luminescence) display, or the like can be used.

  The input unit 205 detects a user input operation and notifies the processor 201 as an input signal. The input operation includes a touch operation in which the touch panel is operated with a pointing device such as a touch pen or a user's finger, an operation in which a plurality of input keys are pressed, and any of these operations may be adopted.

  The input interface 206 is connected to the sensor 11. The sensor 11 is a sensor device that is fixed to the headphones 12 and converts a signal indicating the user's line-of-sight direction into data and outputs the data to the input interface 206. The communication means with the input interface 206 may be wired or wireless. The input interface 206 notifies the processor 201 of the signal acquired from the sensor 11.

  The output interface 207 is connected to the headphones 12, converts an audio signal of a predetermined channel including a synchronization frame or a data frame into an analog audio signal, and outputs the analog audio signal to the headphones 12.

  The wireless interface 208 performs wireless communication with the access points 21a to 21d. The wireless interface 208 has a function of performing demodulation / decoding of a reception signal, encoding / modulation of a transmission signal, and the like.

  FIG. 5 is a block diagram illustrating an example of functions of the voice processing apparatus. The audio processing apparatus 100 includes an arrangement management information storage unit 110, a sound source information storage unit 120, an HRTF information storage unit 130, a user information acquisition unit 140, a virtual speaker arrangement unit 150, a sound source distribution unit 160, and an audio generation unit 170.

  The arrangement management information storage unit 110 temporarily stores information for managing the arrangement of the virtual speakers, such as the user status, information on the sound source, information on the position of the virtual speaker to be arranged, and information on the arrangement area in which the virtual speaker is arranged. Remember. The virtual speaker is a virtual sound source that is virtually arranged on a predetermined circumference centered on the user. Hereinafter, a predetermined circumference centering on the user may be referred to as a “predetermined circumference”.

  In the sound source information storage unit 120, information on each sound source virtually arranged in the vicinity of the user is registered in advance. The information on the sound source includes a sound source ID (Identity) for identifying each sound source, position information on each sound source, and an audio signal output from each sound source. Information about each sound source can be arbitrarily set by an input operation to the sound processing apparatus 100 by an administrator or the like, for example.

  Note that both the above-mentioned “virtual speaker” and “sound source” are virtual sound sources that are virtually arranged around the user. In the following description, a virtual sound source whose position information is preset in the sound source information storage unit 120 is simply referred to as “sound source”, and a virtual sound source dynamically arranged by the virtual speaker arrangement unit 150 is referred to as “virtual speaker”. .

  The HRTF information storage unit 130 stores left and right HRTF list information corresponding to the position of the virtual speaker. HRTF means a transfer function between a sound wave emitted from a sound source having an arbitrary position and a sound wave reaching the eardrum of the ear, and its value varies depending on the direction and altitude of the sound source as viewed from the listener. The sound processing apparatus 100 can localize the sound image in the specific direction by performing a convolution operation on the sound signal using the HRTF in the specific direction.

  The user information acquisition unit 140 acquires information indicating the user's line-of-sight direction from the user terminal 200 as needed. Moreover, the user information acquisition part 140 receives the signal transmitted from the user terminal 200 through the access points 21a-21d, and detects the position of the user terminal 200 based on these received signals. The user information acquisition unit 140 temporarily stores, in the arrangement management information storage unit 110, information indicating the user state such as the acquired user's line-of-sight direction and the detected position information of the user terminal 200.

  Based on the information indicating the state of the user stored in the arrangement management information storage unit 110 and the position information of the sound source stored in the sound source information storage unit 120, the virtual speaker arrangement unit 150 places a certain number of virtual speakers around the user. Are arranged on a predetermined circumference. Then, the virtual speaker placement unit 150 stores information on the placed virtual speakers in the placement management information storage unit 110.

  The sound source distribution unit 160, based on the information indicating the state of the user stored in the arrangement management information storage unit 110 and the information on the position of the virtual speaker, the audio signal corresponding to each sound source stored in the sound source information storage unit 120 Is distributed to each virtual speaker arranged by the virtual speaker arrangement unit 150.

  The sound generation unit 170 acquires the left and right HRTFs corresponding to the positions of the virtual speakers arranged by the virtual speaker arrangement unit 150 based on the HRTF list information stored in the HRTF information storage unit 130. Then, the sound generation unit 170 generates left and right channel sound signals based on the sound signals distributed by the sound source distribution unit 160 and the acquired left and right HRTFs. In addition, the sound generation unit 170 transmits the generated sound signals of the left and right channels to the user terminal 200.

On the other hand, the user terminal 200 includes a user information providing unit 210 and an audio output unit 220.
The user information providing unit 210 calculates the user's line-of-sight direction based on the detection result of the sensor 11, and transmits information indicating the line-of-sight direction to the sound processing apparatus 100. Information indicating the user's line-of-sight direction is transmitted whenever the detection result is output from the sensor 11. The audio output unit 220 converts the audio signal received from the audio processing device 100 into an analog signal, amplifies the analog audio signal, and transmits the amplified analog audio signal to the headphones 12.

  Next, a method for generating left and right channel audio signals from audio signals output from the sound source in a state where the sound source and the virtual speaker are arranged will be described. In the following description, the coordinates in the vertical direction (z-axis direction) are ignored, the user's line-of-sight direction is parallel to the horizontal direction (x-axis direction and y-axis direction), and the height of the sound source and the virtual speaker is high. The height (coordinate in the z-axis direction) is considered to be the same as the height of the user's head (precisely, the ear).

  FIG. 6 is a diagram illustrating an example of generating audio signals of the left and right channels. The user 30 is a person who listens to the voice using the voice processing system of the present embodiment. The sound source 31 is one of sound sources included in m sound sources (m is an integer of 1 or more) arranged around the user 30 in advance. The virtual speakers 32 and 33 are virtual speakers included in n (n is an integer of 1 or more and smaller than m) virtual speakers arranged by the virtual speaker arrangement unit 150. On the circumference 34 centered on the user 30, the virtual speaker 32 (V 0) is arranged at a position moved θv 0 ° in the clockwise rotation direction from the line-of-sight direction 35 of the user 30, and the virtual speaker 33 (V 1) It is arranged at a position moved from the line-of-sight direction 35 by θv1 ° in the clockwise rotation direction.

  The sound source distribution unit 160 distributes the sound signal of each sound source to one or two virtual speakers adjacent to each sound source in the direction along the circumference 34 when viewed from the user 30. For example, the sound signal output from the sound source 31 is generated by the sound source distribution unit 160 with respect to the direction along the circumference 34 when viewed from the user 30 among the plurality of virtual speakers arranged by the virtual speaker arrangement unit 150. Are distributed to the virtual speakers 32 and 33 adjacent to.

  When the sound signal of the sound source is distributed to the two virtual speakers, the sound signal weighted according to the positional relationship between the user, the sound source, and the two virtual speakers is distributed to each virtual speaker. For example, in the example of FIG. 6, an angle formed by a straight line connecting the user 30 and the sound source 31 and a straight line connecting the user 30 and the virtual speaker 32 (V0) is θp, and a straight line connecting the user 30 and the sound source 31 An angle formed by a straight line connecting the user 30 and the virtual speaker 33 (V1) is defined as θq. In this case, the sound source distribution unit 160 uses the sound signal obtained by multiplying the sound signal of the sound source 31 by the weighting coefficient {θq / (θp + θq)} and the weighting coefficient according to the distance between the user 30 and the sound source 31 as a virtual speaker. Distribute to 32 (V0). Further, the sound source distribution unit 160 multiplies the sound signal of the sound source 31 by the weight signal {θp / (θp + θq)} and the weight coefficient corresponding to the distance between the user 30 and the sound source 31 to the virtual speaker 33. Distribute to (V1). Here, the weighting coefficient corresponding to the distance between the user 30 and the sound source 31 is set smaller as the distance increases.

  As shown in FIG. 6, the audio generation unit 170 shown in FIG. 5 includes an audio generation unit 171 for outputting a right channel audio signal and an audio generation unit 172 for outputting a left channel audio signal. Have.

  For example, the sound generation unit (right) 171 performs a convolution operation on the sound signal distributed to the virtual speaker 32 (V0) and the HRTF of the right channel corresponding to θv0 ° indicating the position where the virtual speaker 32 is disposed. Similarly, the sound generation unit (right) 171 outputs the sound signal distributed to the virtual speaker 33 (V1) and the HRTF of the right channel corresponding to θv1 ° indicating the position where the virtual speaker 33 (V1) is disposed. Convolution operation. Then, a right channel audio signal is generated by synthesizing the signals subjected to the convolution operation for each sound source as described above.

  Similarly, the audio generation unit (left) 172 combines the audio signal distributed to each of the n virtual speakers and the left channel HRTF corresponding to the position of each virtual speaker, thereby synthesizing the left channel audio signal. Is generated.

  In this way, by distributing the audio signals from each of a plurality of sound sources to a plurality of virtual speakers that are fewer than the number of these sound sources, generating the sound signals of the left and right channels, the number of convolution operations can be reduced by the number of sound sources. The processing load of the voice generation unit 170 can be reduced. Therefore, the left and right channel audio signals can be generated quickly, and the real-time performance of the audio output can be maintained.

  However, in the method of distributing the sound signal of the sound source to a smaller number of virtual speakers as described above, since the positions of the sound source and the virtual speakers are different, the HRTFs corresponding to the sound sources and the virtual speakers are also different. For this reason, the sound signal generated using the sound signal distributed to the virtual speaker as described above may reduce the sense of localization of the sound image of each sound source. For example, the audio signal of the sound source 31 in FIG. 6 is not processed using the HRTF corresponding to the position of the sound source 31, and the virtual speakers 32 (V0) and 33 (V1) at positions different from the sound source 31 are processed. Processing is performed using the HRTF corresponding to each position. In this case, since calculation using an HRTF different from the HRTF that should be used is performed, the user 30 may not be able to recognize the sound image of the sound source 31 in the correct direction.

7 to 13, a method for arranging virtual speakers so as to improve the sense of localization of the sound images of the sound sources arranged around the user will be described.
FIG. 7 is a diagram illustrating an example of a positional relationship between a sound source and a virtual speaker. Around the user 30, a plurality of sound sources including sound sources 41a and 41b are arranged. The sound source 41a is adjacent to the virtual speaker 42a and the virtual speaker 42b in the direction along the circumference 34, and the sound source 41b is adjacent to the virtual speaker 42b and the virtual speaker 42c in the direction along the circumference 34. Yes.

  The angle θa1 is an angle between the direction from the user 30 to the virtual speaker 42a and the direction from the user 30 to the sound source 41a, and the angle θa2 is the direction from the user 30 to the virtual speaker 42b and from the user 30 to the sound source 41a. Is the angle between Similarly, the angle θb1 is an angle between the direction from the user 30 to the virtual speaker 42b and the direction from the user 30 to the sound source 41b, and the angle θb2 is the direction from the user 30 to the virtual speaker 42c and from the user 30. It is an angle between the direction to the sound source 41b.

  Here, when an audio signal is generated using a virtual speaker, an audio signal closer to the localization of the sound image of the sound source can be generated as the angle between the direction of the virtual speaker viewed from the user 30 and the direction of the sound source is smaller. . Therefore, when generating audio signals using the virtual speakers 42a, 42b, and 42c, when each virtual speaker is arranged so that the maximum values of the angles θa1, θa2, θb1, and θb2 are minimized, The localization of the sound image is improved.

  As described above, the virtual speaker placement unit 150 reduces the distance in the direction along the circumference 34 between each of the two adjacent virtual speakers and the sound source positioned between the two virtual speakers when viewed from the user 30. Thus, each virtual speaker is arranged. This reduces the error between the HRTF that should be used originally and the HRTF used in the operation. Therefore, when the left and right channel audio signals generated using the audio signal output from the virtual speaker are generated, it is possible to improve the sense of localization of the sound image of each sound source.

The distance between the sound source and the user is interpolated by the sound source volume by the sound source distribution unit 160 and the like, and thus does not affect the sense of localization of the sound image.
Next, with reference to FIG. 8, a method for arranging virtual speakers when the number of virtual speakers is equal to or greater than the number of sound sources will be described.

  FIG. 8 is a diagram illustrating an example of a virtual speaker arrangement method when the number of virtual speakers is equal to or greater than the number of virtual sound sources. Sound sources 43 a and 43 b are arranged around the user 30. At this time, when the virtual speakers 51, 52, 53, 54, 55, 56, 57, 58 are arranged on the circumference 34, first, the virtual speaker arrangement unit 150 first displays the virtual speakers 51, 52, 53, 54, 55, 56, 57, 58 are arranged on the circumference 34 so as to be even. Next, the virtual speaker placement unit 150 places the virtual speaker having the shortest distance in the direction along the circumference 34 with the sound source when viewed from the user 30 on the circumference 34 so as to be in the direction from the user 30 to the sound source. Move along.

  For example, the virtual speaker arrangement unit 150 arranges the virtual speaker 52 having the shortest distance in the direction along the circumference 34 with respect to the sound source 43a on the circumference 34 so as to be in the direction from the user 30 to the sound source 43a. Further, the virtual speaker arrangement unit 150 arranges the virtual speaker 54 having the shortest distance in the direction along the circumference 34 with respect to the sound source 43b on the circumference 34 so as to be in the direction from the user 30 to the sound source 43b.

  Thereby, the same HRTF as the sound source 43a is applied to the HRTF of the virtual speaker 52, and the same HRTF as the sound source 43b is applied to the HRTF of the virtual speaker 54. Therefore, the sound generation unit 170 can generate a sound signal in which each sound image of the sound sources 43a and 43b is accurately localized in each direction.

  Thus, when the number of virtual speakers is equal to or greater than the number of sound sources, the same HRTF as each sound source can be applied by arranging virtual speakers from the user 30 in the direction of each sound source on the circumference 34. The sound image of the sound source can be accurately localized in the direction of each sound source.

  For the virtual speakers other than the moved virtual speakers 52 and 54, the sound signal output from the sound sources 41 and 42 may not be distributed by the sound source distribution unit 160. Further, the virtual speaker placement unit 150 may not place virtual speakers other than the moved virtual speakers 52 and 54.

Next, with reference to FIGS. 9 to 13, a method for arranging virtual speakers when the number of virtual speakers is less than the number of sound sources will be described.
FIG. 9 is a first diagram illustrating an example of an arrangement method of virtual speakers when the number of virtual speakers is less than the number of virtual sound sources. In the vicinity of the user 30, 16 sound sources of sound source 61 (sound source ID = # 1) to sound source 76 (sound source ID = # 16) are arranged (in the following description, “sound source ID =” is described. (Omitted). In this case, a case where eight virtual speakers are arranged will be described. Hereinafter, the same applies to FIGS. 10 to 13.

First, the virtual speaker arrangement unit 150 calculates an angle in a direction along the circumference around the user 30 between two sound sources adjacent in the direction along the circumference 34 when viewed from the user 30.
Hereinafter, for example, an angle in a direction along the circumference 34 around the user 30 between the sound source 61 (# 1) and the sound source 62 (# 2) is represented as Sθ1-2, and the sound source 62 (# 2). The angle between the sound source 63 and the sound source 63 (# 3) is represented as Sθ2-3. The angle in the direction along the circumference around the user 30 between the other two sound sources is also expressed in the same manner.

  FIG. 10 is a second diagram illustrating an example of a placement method of virtual speakers when the number of virtual speakers is less than the number of virtual sound sources. Next, the virtual speaker placement unit 150 selects a pair of sound sources in which the angle between the sound sources calculated in FIG. 9 is equal to or greater than a threshold value in the range of directions along the circumference 34, and sets the selected pair. Each included sound source is identified as a “both-end sound source”. This “both-end sound source” means that the sound source is located in the direction of both ends of an area that is a candidate for an “arrangement area” to be described later when viewed from the user 30.

  For example, the threshold value is set to a value obtained by dividing 360 ° by the number of virtual speakers “8” (that is, 45 °). For example, if only the angles Sθ4-5 and Sθ8-9 are 45 ° or more, the sound source 64 (# 4), the sound source 65 (# 5), the sound source 68 ( # 8) and the sound source 69 (# 9) are specified as the sound sources at both ends by the virtual speaker placement unit 150.

The threshold value may be smaller than a value obtained by dividing 360 ° by the number of virtual speakers.
Next, the virtual speaker placement unit 150 places virtual speakers in the direction in which the identified both-end sound sources are located. In the case of FIG. 10, the virtual speaker placement unit 150 places the virtual speaker 81 in the direction of the sound source 64 (# 4), places the virtual speaker 82 in the direction of the sound source 65 (# 5), and the sound source 68 (# 8). The virtual speaker 83 is disposed in the direction of the sound source 69, and the virtual speaker 84 is disposed in the direction of the sound source 69 (# 9).

  Next, the virtual speaker placement unit 150 selects a virtual speaker as an area that further includes sound sources other than both-end sound sources among areas sandwiched between both-end sound sources adjacent to each other in the direction of the circumference 34 when viewed from the user 30. Further, it is specified as an “arrangement area” for arrangement.

  In the case of FIG. 10, the region Tθ5-8 sandwiched between the sound source 65 (# 5) and the sound source 68 (# 8) includes the sound source 66 (# 6) and the sound source 67 (# 7). Further, in a region Tθ9-4 sandwiched between the sound source 69 (# 9) and the sound source 64 (# 4), the sound source 61 (# 1) to the sound source 63 (# 3), the sound source 70 (# 10) to the sound source 76 (# 16). Therefore, the region Tθ5-8 and the region Tθ9-4 are specified as the arrangement region.

  FIG. 11 is a third diagram illustrating an example of a placement method of virtual speakers when the number of virtual speakers is less than the number of virtual sound sources. Next, the virtual speaker placement unit 150 calculates an angle in a direction along the circumference 34 between two adjacent virtual speakers that have already been placed in each of the specified placement regions. For example, in the case of FIG. 11, the angle in the direction along the circumference 34 between the two adjacent virtual speakers 82 and 83 in the region Tθ5-8 is an angle Sθ5-8. In the region Tθ9-4, the angle in the direction along the circumference 34 between the two adjacent virtual speakers 84 and 81 is the angle Sθ9-4.

  Next, the virtual speaker placement unit 150 places one unplaced virtual speaker in the placement region where the calculated angle between the virtual speakers is maximum so that the intervals between the virtual speakers are equal. In FIG. 11, when the angle Sθ5-8 <angle Sθ9-4, the virtual speaker arrangement unit 150 arranges the virtual speakers 85 in the region Tθ9-4 so that the intervals between the virtual speakers are equal.

  FIG. 12 is a fourth diagram illustrating an example of a placement method of virtual speakers when the number of virtual speakers is less than the number of virtual sound sources. Next, the virtual speaker arrangement unit 150 calculates the angle in the direction along the circumference 34 between two adjacent virtual speakers that have already been arranged in each of the specified arrangement areas, as in FIG. The virtual speaker is arranged in the arrangement region where the angle formed is the maximum. In FIG. 12, the angle Sθ5-8 is calculated in the region Tθ5-8, and (angle Sθ9-4) / 2 is calculated in the region Tθ9-4. When the angle Sθ5-8> (angle Sθ9-4) / 2, the virtual speakers 86 are arranged in the region Tθ5-8 so that the distances between the virtual speakers are equal.

  FIG. 13 is a fifth diagram illustrating an example of a method of arranging virtual speakers when the number of virtual speakers is less than the number of virtual sound sources. Next, the virtual speaker placement unit 150 calculates the angle in the direction along the circumference 34 between two adjacent virtual speakers that have already been placed in each of the specified placement regions, as in FIGS. 11 and 12. Then, the virtual speaker is arranged in the arrangement area where the calculated angle is the maximum. In FIG. 13, (angle Sθ5-8) / 2 is calculated in the region Tθ5-8, and (angle Sθ9-4) / 2 is calculated in the region Tθ9-4. If (angle Sθ5-8) / 2 <(angle Sθ9-4) / 2, virtual speakers 87 are arranged in region Tθ9-4 so that the distances between the virtual speakers are equal. Hereinafter, the process of FIG. 13 is repeated until there is no virtual speaker that is not arranged.

  As shown in FIGS. 9 to 13, the virtual speaker placement unit 150 first sets the angle in the direction along the circumference 34 between two sound sources adjacent to each other in the direction along the circumference 34 when viewed from the user 30. Based on this, the arrangement area is specified. Then, in each of the specified arrangement areas, the virtual speakers are arranged in the arrangement area where the angle in the direction along the circumference 34 between the two adjacent virtual speakers is the maximum, until there is no unplaced virtual speaker. repeat.

  Thereby, each virtual speaker is arranged so that the distance in the direction along the circumference 34 between each of the two adjacent virtual speakers and the sound source located between the two virtual speakers when viewed from the user 30 is reduced. it can. Therefore, it is possible to reduce an error between the HRTF to be originally used and the HRTF used in the operation.

Next, an example of table information used in the processing of the voice processing device 100 will be described.
FIG. 14 is a diagram illustrating an example of a user status table. The user status table 111 is a table that temporarily stores information indicating the user status acquired by the user information acquisition unit 140. The user status table 111 is stored in the arrangement management information storage unit 110. The user status table 111 is updated as needed each time information indicating the user's line-of-sight direction is received from the user terminal 200 or user position information is detected. The user status table 111 includes items of user ID, coordinates, and head posture angle.

An identifier for identifying the user is set in the user ID item.
In the coordinate item, coordinates of the position of the user's head are set. In FIG. 14, this item is set in three dimensions, but may be set in two dimensions. The same applies to the coordinates described below.

In the head posture angle item, an angle of the user's line-of-sight direction with respect to a predetermined reference direction (for example, the north direction) is set. The reference direction is a direction along the horizontal direction.
FIG. 15 is a diagram illustrating an example of a sound source management table. The sound source management table 112 is a table that stores information regarding sound sources arranged around the user. The sound source management table 112 temporarily stores information on sound sources in the order in which they exist in a predetermined direction (for example, the right rotation direction) along the circumference centered on the user from the user's line-of-sight direction. The sound source management table 112 is stored in the arrangement management information storage unit 110. The sound source management table 112 has items of user ID, sound source ID, sound source position, and both end flags.

An identifier for identifying the user is set in the user ID item.
In the sound source ID item, an identifier for identifying the sound source is set.
In the sound source position item, information indicating the position of the sound source is set. The information indicating the position of the sound source may be, for example, relative coordinates of the sound source based on the user's position and line-of-sight direction, or the latitude and longitude of the sound source. The information indicating the position of the sound source may be an angle between the user's line-of-sight direction and the direction of the sound source viewed from the user.

  Information indicating whether the sound source is a both-end sound source is set in the both-end flag item. For example, “TRUE” is set when the sound source is a double-ended sound source, and “FALSE” is set when the sound source is not a double-ended sound source. The initial value of the both-end flag is “FALSE”.

  Although not shown, in the sound source information storage unit 120, the position information of each sound source is registered in advance in association with the sound source ID, for example, by an administrator's setting operation, and the sound source ID and the sound source position of the sound source management table 112 are registered. In each item, a sound source ID and position information of each sound source registered in the sound source information storage unit 120 are set. The order of registration of sound sources in the sound source management table 112 is updated by the user information acquisition unit 140 every time the corresponding user position moves.

  FIG. 16 is a diagram illustrating an example of the virtual speaker position table. The virtual speaker position table 113 is a table that stores information related to the position of the virtual speaker. The virtual speaker position table 113 is temporarily stored in the arrangement management information storage unit 110. The virtual speaker position table 113 includes items of a user ID, a virtual speaker ID, a speaker position, and an arrangement determination flag.

An identifier for identifying the user is set in the user ID item.
In the virtual speaker ID item, an identifier for identifying the virtual speaker is set.
Information indicating the position of the virtual speaker is set in the speaker position item. The information indicating the position of the virtual speaker may be, for example, relative coordinates of the virtual speaker based on the user's position and line-of-sight direction, or may be absolute coordinates of the virtual speaker. The information indicating the position of the speaker may be an angle between the direction of the user and the direction of the virtual speaker viewed from the user.

  In the item of the placement confirmation flag, information indicating whether or not the position of the virtual speaker to be placed is confirmed is set. For example, “TRUE” is set when the position of the virtual speaker is fixed, and “FALSE” is set when the position of the virtual speaker is not fixed. The initial value of the placement confirmation flag is “FALSE”.

  FIG. 17 is a diagram illustrating an example of arrangement information. The arrangement information 114 is information regarding the number of virtual speakers to be arranged. The arrangement information 114 is temporarily stored in the arrangement management information storage unit 110. The placement information 114 includes items of user ID, placed, unplaced, and total.

An identifier for identifying the user is set in the user ID item.
In the arranged item, the number of arranged virtual speakers is set. The placed initial value is “0”.

In the unplaced item, the number of virtual speakers that are not placed is set. The unallocated initial value is the same value as the total item.
In the total item, the total number of virtual speakers to be arranged is set. That is, the total of the arranged virtual speakers and the unplaced virtual speakers is set.

  FIG. 18 is a diagram illustrating an example of an arrangement area management table. The arrangement area management table 115 is a table that stores information related to an arrangement area in which virtual speakers are arranged. The arrangement area management table 115 is temporarily stored in the arrangement management information storage unit 110. The arrangement area management table 115 has items of user ID, area ID, both-end angle, search flag, angle (after division), and number of divisions.

An identifier for identifying the user is set in the user ID item.
An identifier for identifying the arrangement area is set in the area ID item.
In the both end angle item, an angle in a direction along the circumference with the user at the center between both ends of the arrangement area is set.

  In the search flag item, information indicating whether or not it is an arrangement region for adding one virtual speaker is set. For example, “TRUE” is set in the case where the virtual speaker is added, and “FALSE” is set in the case where the virtual speaker is not added. Whether or not it is a placement area for adding a virtual speaker is whether or not the angle in the direction along the circumference centered on the user between the virtual speakers evenly placed in each placement area is the maximum compared to other placement areas. Is judged.

  In the item of angle (after division), an angle in a direction along the circumference around the user is set between the virtual speakers arranged uniformly in the arrangement region. Specifically, in the angle (after division) item, a numerical value obtained by dividing the angle set in the both-end angle item by the value set in the division number item is set. In the item of the number of divisions, the number of the arrangement area divided by the virtual speakers excluding both ends thereof is set, and specifically, a value calculated as the number of virtual speakers arranged in the arrangement area minus 1 is set. Is done.

The area ID, the both-end angle, the search flag, the angle (after division), and the initial value of the number of divisions are blank.
Next, processing of the voice processing apparatus 100 will be described using a flowchart.

  FIG. 19 is a flowchart illustrating an example of virtual speaker arrangement processing. In the processing of FIG. 19, it is assumed that the sound source information storage unit 120 stores the position information of the sound source and the audio signal output by the sound source. Moreover, the process of FIG. 19 shall be performed for every user. Accordingly, in the process of FIG. 19 (and FIG. 20), a table in which a user ID corresponding to a user to be processed is registered as each table including a user ID item such as the user status table 111.

  (Step S11) The user information acquisition unit 140 acquires information indicating the state of the user. The information indicating the user state includes information indicating the user's line-of-sight direction and user position information.

  The user information acquisition unit 140 receives information indicating the user's line-of-sight direction from the user terminal 200. Further, for information indicating the position of the user terminal 200, the user information acquisition unit 140 uses trigonometry based on a difference in signal reception time or a difference in received radio wave intensity at the access points 21a to 21d. The position of is detected.

  The user information acquisition unit 140 temporarily stores information indicating the acquired user status in the user status table 111. At this time, the user information acquisition unit 140 sets information indicating the position of the user terminal 200 in the coordinate item, and sets the user's line-of-sight direction and a predetermined reference direction (for example, the north direction) in the head posture angle item. Set the angle between.

  (Step S <b> 12) The virtual speaker arrangement unit 150 reads information related to each sound source from the sound source information storage unit 120. The information on the sound source includes a sound source ID for identifying the sound source, position information of the sound source, and an audio signal output from the sound source.

  (Step S <b> 13) The virtual speaker placement unit 150 displays information on the sound source based on the information indicating the user's line-of-sight direction and position stored in the user state table 111 and the position information of each sound source confirmed in Step S <b> 12. 112.

  Specifically, first, the virtual speaker placement unit 150 reads information indicating the user state from the user state table 111. Next, based on the read user information and the coordinates of the sound source confirmed in step S12, the virtual speaker placement unit 150 moves from the user's line-of-sight direction to a circumference centered on the user (hereinafter, a predetermined circumference). The sound sources that exist in the right rotation direction are sequentially determined, and information about each sound source is registered in the sound source management table 112 in the determination order. At that time, the virtual speaker arrangement unit 150 sets the position information of the sound source confirmed in step S12 in the sound source position item, and sets “FALSE” as the initial value in the both-end sound source flag item.

  (Step S14) The virtual speaker placement unit 150 determines whether the number of virtual speakers to be placed is less than the number of sound sources stored in step S13. If the number of virtual speakers to be arranged is less than the number of stored sound sources, the process proceeds to step S21. If the number of virtual speakers to be arranged is equal to or greater than the number of stored sound sources, the process proceeds to step S15.

  (Step S15) The virtual speaker placement unit 150 places virtual speakers on a straight line connecting the user and the sound source for each sound source on a predetermined circumference. At this time, the virtual speaker placement unit 150 registers information about each placed virtual speaker in the virtual speaker position table 113. At this time, the virtual speaker arrangement unit 150 sets the position information of the virtual speaker arranged in the item of the speaker position, and sets “TRUE” in the item of the arrangement confirmation flag. The position information of the virtual speaker is calculated based on the positional relationship between the coordinates of the user acquired in step S11, the radius of the predetermined circumference, and the coordinates of the sound source arranged in the same direction as the virtual speaker as viewed from the user. The

Information about a predetermined circumference (that is, the distance between the user and the virtual speaker) is stored in advance in a storage area such as the HDD 103, for example.
(Step S16) The sound source distribution unit 160 distributes the audio signal output from each sound source stored in the sound source information storage unit 120 to two adjacent virtual speakers along a predetermined circumference. Specifically, as described with reference to FIG. 6, the audio signal distributed to the virtual speakers is generated by weighting according to the positional relationship between the user, the sound source, and the two virtual speakers.

(Step S <b> 17) The sound generation unit 170 generates left and right channel sound signals using the distributed sound signals.
Specifically, first, the sound generation unit 170 determines between the user's line-of-sight direction and the direction in which the virtual speaker is arranged based on the speaker position of each virtual speaker registered in the virtual speaker position table 113. Calculate the angle. Next, the voice generation unit 170 searches the HRTF information storage unit 130 for left and right HRTFs that match the calculated angles. Then, the audio generation unit 170 combines the audio signals distributed by the sound source distribution unit 160 and the searched left and right HRTFs to generate audio signals for the left and right channels.

  As an example of the convolution operation, in the audio signal from time τ to time t, the audio signal of the left and right channels is h (t), the function of the distributed audio signal is f (t), and HRTF is g (t) In this case, the following convolution integral can be used.

Then, the sound generation unit 170 transmits the generated sound signals of the left and right channels to the user terminal 200.
Thereafter, the audio output unit 220 of the user terminal 200 receives the audio signals of the left and right channels from the audio processing device 100. The audio output unit 220 converts the received audio signal into an analog audio signal, and outputs the converted analog audio signal to the headphones 12.

FIG. 20 is a flowchart (continued) illustrating an example of the virtual speaker arrangement process.
(Step S <b> 21) The virtual speaker placement unit 150 identifies both-end sound sources as described with reference to FIGS. 9 to 10.

  Specifically, first, the virtual speaker arrangement unit 150 acquires information regarding a plurality of sound sources arranged around the user from the sound source management table 112. Next, based on the sound source position of each acquired sound source, the virtual speaker placement unit 150, as described in FIG. 9, between two sound sources adjacent in a direction along a predetermined circumference as viewed from the user, The angle in the direction along the circumference is calculated. Next, as described with reference to FIG. 10, a sound source pair in which the calculated angle between the sound sources is equal to or greater than a threshold value is specified in a range of directions along a predetermined circumference. Then, the virtual speaker arrangement unit 150 identifies the sound sources included in the identified sound source pair as both-end sound sources, and updates the both-end flag item of the sound source to “TRUE” in the sound source management table 112.

  (Step S22) As described with reference to FIG. 10, the virtual speaker placement unit 150 places virtual speakers in the direction in which the sound sources at both ends are located on a predetermined circumference. At this time, the virtual speaker arrangement unit 150 stores information on the arranged virtual speakers in the virtual speaker position table 113, as in step S15 of FIG.

  Then, the virtual speaker placement unit 150 updates the value of the placed item in the placement information 114 to a value obtained by adding the number of stored virtual speakers, and changes the value of the unplaced item to the number of stored virtual speakers. Update to the value obtained by subtracting.

In subsequent steps S23 to S27, processing for arranging the remaining virtual speakers excluding those arranged by the initial arrangement is performed.
(Step S23) The virtual speaker placement unit 150 identifies, as a placement area, a range in which a sound source is present at positions other than both ends of the plurality of ranges sandwiched between the virtual speakers already placed in Step S22. To do.

  Specifically, the virtual speaker arrangement unit 150 refers to the sound source management table 112 and includes only sound sources having both ends flag “FALSE” between both ends sound sources having both ends flag “TRUE”. An area is specified as an arrangement area.

  Next, the virtual speaker placement unit 150 registers information regarding the identified placement area in the placement area management table 115. At this time, the virtual speaker placement unit 150 sets the angle between the two sound sources as viewed from the user in the both end angle and angle (after division) items, and sets “FALSE” in the search flag item. , “1” is set as the number of divisions.

  (Step S24) As described with reference to FIGS. 11 to 13, the virtual speaker placement unit 150 selects a placement area in which one virtual speaker is added from the placement area specified in step S23.

  Specifically, the virtual speaker arrangement unit 150 selects an arrangement area having a maximum angle (after division) value from the arrangement area management table 115. Then, the virtual speaker arrangement unit 150 updates the search flag for the selected arrangement area to “TRUE” in the arrangement area management table 115.

(Step S25) The virtual speaker placement unit 150 places one virtual speaker as described with reference to FIGS.
Specifically, the virtual speaker placement unit 150 updates the placement area management table 115 to a value obtained by adding 1 to the number of divisions of the search area division number where the search flag is “TRUE”. Also, the virtual speaker placement unit 150 updates the value set in the angle (after division) item of the placement region whose search flag is “TRUE” and the value set in the both end angle item in the placement region management table 115 after the update. Update to the value divided by the value set for the number. Further, the virtual speaker placement unit 150 updates the search flag item of the placement area whose search flag is “TRUE” to “FALSE” in the placement area management table 115.

Then, the virtual speaker placement unit 150 updates the placed item of the placement information 114 to a value obtained by adding 1, and updates the unplaced item of the placement information 114 to a value obtained by subtracting 1.
(Step S26) As described with reference to FIGS. 11 to 13, the virtual speaker placement unit 150 determines whether all the virtual speakers have been placed. If all virtual speakers have been arranged, the process proceeds to step S27. If not all virtual speakers have been arranged, the process proceeds to step S24. Whether or not all the virtual speakers have been arranged can be determined, for example, based on whether the unarranged item of the arrangement information 114 is “0”.

  (Step S <b> 27) As described with reference to FIGS. 11 to 13, the virtual speaker placement unit 150 determines a predetermined angle so that the angle between two adjacent virtual speakers is centered on the user for each placement region. The coordinates of the remaining virtual speakers arranged on the circumference are registered in the virtual speaker position table 113.

Specifically, the virtual speaker placement unit 150 sequentially selects placement areas from the placement area management table 115 and executes the following process for each of the selected placement areas.
First, the virtual speaker placement unit 150 reads the angle (after division) for the placement area selected from the placement area management table 115. The virtual speaker arrangement unit 150 arranges the virtual speaker at a position for each read angle (after division) with respect to one direction (for example, clockwise direction) from one end of the selected arrangement area on a predetermined circumference. , The position information (for example, coordinates) of the virtual speakers for “the number of divisions of the selected arrangement area−1” is calculated. Then, the virtual speaker arrangement unit 150 updates the information regarding the virtual speaker to be arranged in the virtual speaker position table 113. Specifically, the virtual speaker placement unit 150 registers the position information calculated in the speaker position item for the virtual speaker whose placement decision flag is “FALSE”, and updates the placement decision flag item to “TRUE”. To do.

Then, the virtual speaker arrangement unit 150 proceeds with the process to step S16.
Next, FIGS. 21 to 25 will specifically describe an example of processing when virtual speakers are arranged as in steps S <b> 21 to S <b> 25 of FIG. 20.

  FIG. 21 is a first diagram illustrating an example of the arrangement of virtual speakers. In step S21 of FIG. 20, sound sources included at both ends of the arrangement area are specified. Here, as in the example of FIG. 10, a pair of sound source ID = # 4, # 5 and a pair of sound source ID = # 8, # 9 are extracted as a pair of sound sources having an angle equal to or greater than a threshold value. Suppose. As a result, sound sources with sound source ID = # 4, # 5, # 8, and # 9 are specified as both-end sound sources.

  Therefore, the virtual speaker arrangement unit 150 updates the sound source both end flag items of the sound source IDs # 4, # 5, # 8, and # 9 to “TRUE” as in the sound source management table 112a.

  FIG. 22 is a second diagram illustrating an example of the arrangement of virtual speakers. In step S22 of FIG. 20, the virtual speaker placement unit 150 has virtual speaker IDs = V1 and V2 in a direction in which each of the sound sources specified in step S21 of FIG. , V3 and V4 are arranged. The virtual speaker arrangement unit 150 includes sound source IDs = # 4, # 5, # 8, # in the items of the speaker positions corresponding to the virtual speaker IDs = V1, V2, V3, and V4 as in the virtual speaker position table 113a. 9, the position information of the virtual speakers respectively arranged in the direction of the sound source is registered, and the item of the corresponding arrangement confirmation flag is updated to “TRUE”.

  Further, in step S23 of FIG. 20, the virtual speaker placement unit 150 selects a region where a sound source is present at a position other than both ends among regions sandwiched between adjacent both end sound sources based on the sound source management table 112a. It is specified as a placement area. As a result, it is assumed that the area IDs = Tθ5-8 and Tθ9-4 are specified as the arrangement areas. In this case, as in the placement area management table 115a, the virtual speaker placement section 150 registers the area ID, the both end angles, the search flag, the angle (after division), and the number of divisions of each placement area.

  Further, as indicated by the virtual speaker position table 113a, the number of virtual speakers whose arrangement confirmation flag is “TRUE” is four. For this reason, although not shown, the arrangement information 114 is also updated from “0” to “4” for the arranged virtual speakers and from “8” to “4” for the non-arranged virtual speakers.

  FIG. 23 is a third diagram illustrating an example of arrangement of virtual speakers. In step S24 of FIG. 20, an arrangement area for adding one virtual speaker is selected from the arrangement area specified in step S23. In step S25 of FIG. 20, one virtual speaker is arranged in the selected area.

  Specifically, first, in step S <b> 24 of FIG. 20, the virtual speaker placement unit 150 selects a placement region where the angle of the placement region (after division) is maximum. Like the arrangement area management table 115a, the both end angle of the arrangement area whose area ID is Tθ9-4 is 80 °, and the both end angle of the arrangement area whose area ID is Tθ5-8 is 30 °. Therefore, the virtual speaker placement unit 150 selects a placement region whose region ID is Tθ9-4 as a region to which one virtual speaker to be placed is added.

  Next, in step S25 of FIG. 20, the virtual speaker arrangement unit 150 updates the search flag for the arrangement area whose selected area ID is Tθ9-4 to “TRUE” as in the arrangement area management table 115b.

  Then, the virtual speaker arrangement unit 150 updates the angle (after division) to the value “2” obtained by adding 1 to the division number in the arrangement region where the search flag is “TRUE” as in the arrangement region management table 115c. The value set to the item of both-ends angle is updated to the value “40” obtained by dividing the value set by the updated division number, and the search flag item is updated to “FALSE”. Then, the virtual speaker placement unit 150 updates the value “5” obtained by adding 1 to the placed virtual speaker, and updates the value “3” obtained by subtracting one unplaced virtual speaker, as in the placement information 114b. .

  FIG. 24 is a fourth diagram illustrating an example of the arrangement of virtual speakers. Next, the virtual speaker placement unit 150 performs the same processing as in FIG. 23 in each placement region. As shown in the arrangement area management table 115c, the area ID of the arrangement area having the maximum angle (after division) is Tθ9-4.

  Therefore, the virtual speaker arrangement unit 150 updates the number of divisions of the arrangement area whose area ID is Tθ9-4 to the value “3” added by 1 as in the arrangement area management table 115d, and the area ID is Tθ9-4. The angle (after division) of a certain arrangement area is updated to both-end angle / updated division number “26”. Also, the virtual speaker placement unit 150 updates the placed virtual speaker to a value “6” obtained by adding 1 and updates the unassigned virtual speaker to a value “2” obtained by subtracting 1 as shown in the placement information 114c. .

  Next, the virtual speaker placement unit 150 executes the same processing as in FIG. 23 for each placement region after the virtual speaker is added. As shown in the arrangement area management table 115d, the area ID of the arrangement area having the maximum angle (after division) is Tθ5-8.

  Therefore, the virtual speaker arrangement unit 150 updates the number of divisions of the arrangement area whose area ID is Tθ5-8 to “2” added by 1 as in the arrangement area management table 115e, and the area ID is Tθ5-8. The angle (after division) of a certain arrangement area is updated to both-end angle / updated division number “15”. In addition, the virtual speaker placement unit 150 updates the placed virtual speaker to a value “7” obtained by adding 1 and updates the unassigned virtual speaker to a value “1” obtained by subtracting 1 as the placement information 114d. .

  FIG. 25 is a fifth diagram illustrating an example of arrangement of virtual speakers. For each undistributed virtual speaker, as shown in FIGS. 23 to 24, after calculating the placement area to be placed, in step S <b> 27 of FIG. 20, the virtual speaker placement unit 150 sets two adjacent loudspeakers for each placement area. The added virtual speakers are arranged on a predetermined circumference so that the angles between the virtual speakers around the user 30 are equal. The angle between the virtual speakers arranged in each arrangement area is a value of an angle (after division) associated with each arrangement area in the arrangement area management table 115.

  As a result, as in the virtual speaker position table 113b, the coordinates of the virtual speakers newly arranged in FIG. 23 to FIG. 24 and the placement confirmation flags of the virtual speakers with V = V5, V6, V7, and V8 are updated.

  According to the sound processing system of the second embodiment, the sound processing device 100 is configured to have a predetermined circle centered on the user 30 between two sound sources adjacent to each other in a direction along a predetermined circumference as viewed from the user 30. The angle in the direction along the circumference is calculated. The sound processing device 100 identifies a virtual sound source whose calculated angle is equal to or greater than a threshold value as a both-end sound source.

  Next, as viewed from the user 30, virtual speakers are arranged in the direction in which each identified both-end sound source is located. Then, on a predetermined circumference, a region where a sound source exists at positions other than both ends of the region sandwiched between already arranged virtual speakers (ie, a region sandwiched between both end sound sources) is arranged. Identify the area.

  Then, in each of the specified placement areas, 1 virtual speaker that has not been placed is arranged so that the spacing between the virtual speakers is uniform in the placement area where the angle between the two adjacent virtual speakers around the user 30 is the maximum. The process of arranging one is repeated until there are no virtual speakers that are not arranged.

  Thereby, the angle between the straight line connecting the virtual speaker and the user 30 and the straight line connecting the sound source adjacent to the virtual speaker and the user 30 can be reduced. Here, when the audio signal output from the sound source is distributed to the virtual speakers, the smaller the angle between the straight line connecting the virtual speaker and the user 30 and the straight line connecting the sound source adjacent to the virtual speaker and the user 30 is HRTF. This reduces the error in the sound image and suppresses a decrease in the sense of localization of the sound image. Therefore, by reducing the angle between the straight line connecting the virtual speaker and the user 30 and the straight line connecting the sound source adjacent to the virtual speaker and the user 30, it is possible to suppress a reduction in the localization of the sound image of each sound source. Therefore, it is possible to improve the sense of localization of the sound image while reducing the load of processing for localizing the sound images of the plurality of virtual sound sources.

[Modification of Second Embodiment]
Next, a modification of the second embodiment will be described. The following modification improves the sense of direction of the sound source that exists in the direction of a desired range with respect to the listener. For example, there is a case where it is desired to improve the azimuth feeling of a sound source existing ahead of the listener rather than the azimuth feeling of a sound source existing behind the listener. This is because the human sense of direction tends to be more ambiguous in front than behind. In this case, it can be realized by arranging more virtual speakers in the direction of the front range than the direction of the rear range. Therefore, in FIG. 26 to FIG. 27, an example in which more virtual speakers are arranged in the arrangement area existing in front of the user 30 by weighting according to the position of each arrangement area will be described. 26 to 27, differences from the second embodiment will be described, and description of the same configuration and processing as those of the second embodiment will be omitted.

  FIG. 26 is a diagram illustrating an example of a weight setting method for an arrangement region. The circumference 34 is divided into regions K1, K2, K3, and K4. The area K1 is present in front of the user 30, the area K2 is present on the left side of the user 30, the area K3 is present on the right side of the user 30, and the area K4 is present behind the user. For example, a weight of 1.0 is set for the area K1, a weight of 0.8 is set for the areas K2 and K3, and a weight of 0.6 is set for the area K4. In this way, the weight value is set larger in the front direction than in the rear direction of the user 30.

  Here, it is assumed that the arrangement regions T1 and T2 exist on the circumference 34. Since the direction 35a obtained by equally dividing the angle between the directions of both ends in the arrangement region T1 is included in the region K1, the weight of the arrangement region T1 is 1.0. In addition, since the direction 35b obtained by equally dividing the angle between the two end directions in the arrangement region T2 is included in the region K4, the weight of the arrangement region T2 is 0.6.

  FIG. 27 is a diagram illustrating a modification of the placement of the virtual speakers in consideration of the weight. In the arrangement area management tables 116a and 116b, items of angle (after weighting) and weight are added to the arrangement area management table 115. In the item of angle (after weighting), a value obtained by multiplying the value of angle (after division) by the weight is set. In the item of weight, as described with reference to FIG. 26, a weight corresponding to the position of the arrangement area is set.

Here, an example will be described in which the placement area to which the virtual speaker is added is identified and the virtual speaker is placed in steps S24 to S25 in FIG.
First, in a modified example of the system of the second embodiment, a virtual speaker is added to an arrangement area where the angle (after weighting) is maximum, not the angle (after division) of the arrangement area. In the example of the arrangement area management table 116a, the arrangement area whose area ID is Tθ9-4 is selected as the area to which the virtual speaker is added, and the search flag of the arrangement area is updated to “TRUE”.

  Then, as in the arrangement area management table 116b, the angle (after division), the angle (after weighting), and the division number of the arrangement area whose area ID is Tθ9-4 are updated. In other words, the angle of the arrangement area whose area ID is located behind Tθ9-4 and whose area ID is Tθ5-8 is corrected to be smaller than the original by weighting and used for the calculation. As a result, the virtual speaker is preferentially added toward the arrangement area having the area ID Tθ9-4 positioned further forward, and as a result, the forward orientation feeling is improved.

Note that the number of regions described in FIG. 26 may be set to four or more. Moreover, you may enable it to set the maximum number of the virtual speakers which can be arrange | positioned for every area | region K1, K2, K3, K4.
According to the modification of the second embodiment, the sound processing apparatus 100 places a large number of virtual speakers in a desired direction of the user 30 by weighting the angle between the virtual speakers based on the position of the placement region. can do. If many virtual speakers are arranged, the angle between the direction of the sound source viewed from the user 30 and the direction of the virtual speaker viewed from the user 30 can be reduced. Therefore, the feeling of localization of the sound images of a plurality of sound sources arranged in the direction desired by the user 30 is improved.

  As described above, the information processing according to the first embodiment can be realized by causing the voice processing device 1 to execute a program, and the information processing according to the second embodiment can be performed by the voice processing device 100 or a user terminal. This can be realized by causing the program to be executed by 200. Such a program can be recorded on a computer-readable recording medium (for example, the recording medium 15). As the recording medium, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used. Magnetic disks include FD and HDD. Optical discs include CD, CD-R (Recordable) / RW (Rewritable), DVD, and DVD-R / RW.

  When distributing the program, for example, a portable recording medium in which the program is recorded is provided. It is also possible to store the program in a storage device of another computer and distribute the program via the network 10. The computer stores, for example, a program recorded on a portable recording medium or a program received from another computer in a storage device (for example, HDD 103), and reads and executes the program from the storage device. However, a program read from a portable recording medium may be directly executed, or a program received from another computer via the network 10 may be directly executed. Further, at least a part of the information processing described above can be realized by an electronic circuit such as a DSP, ASIC, or PLD (Programmable Logic Device).

DESCRIPTION OF SYMBOLS 1 Sound processing apparatus 2 Speaker arrangement | positioning part 3 Voice synthesis | combination part 4 Listener 5a, 5b, 5c, 5d Virtual sound source 6a, 6b, 6c Virtual speaker 7 Circumference (theta) 1, (theta) 2, (theta) 3, (theta) 4 angle

Claims (9)

A speaker arrangement unit that arranges a plurality of virtual speakers less than the number of virtual sound sources arranged around the listener on a circumference centered on the listener;
Audio signals from each of the plurality of virtual sound sources are distributed to one or more virtual speakers selected for each virtual sound source among the plurality of virtual speakers, and a head related transfer function corresponding to the position of each virtual speaker is used. A voice synthesizer for synthesizing the voice signal distributed to each virtual speaker into two left and right channel voice signals;
Have
The speaker placement section is
Calculating the angle in the direction along the circumference around the listener, between two virtual sound sources adjacent in the direction along the circumference as seen from the listener;
Of the range of directions along the circumference around the listener, the calculated angle is sandwiched between virtual sound sources that are equal to or greater than a threshold, and the direction of each virtual sound source viewed from the listener Arranging the plurality of virtual speakers in a second range excluding a first range not including the position of
Features and be Ruoto voice processing apparatus that.
  The audio processing apparatus according to claim 1, wherein the threshold value is a value obtained by dividing 360 ° by the number of virtual speakers.   The speaker arrangement unit identifies a pair of virtual sound sources in which the calculated angle is equal to or greater than the threshold value, and places the virtual speakers in a direction in which each virtual sound source included in the pair is located when viewed from the listener The audio processing apparatus according to claim 1, wherein the remaining virtual speakers are distributed in the second range.   In the second range, the speaker placement unit has a direction along the circumference of each of two adjacent virtual speakers and a virtual sound source positioned between the two virtual speakers when viewed from the listener. The audio processing apparatus according to claim 3, wherein the remaining virtual speakers are arranged in the second range so that the distance is reduced.   When a plurality of the pairs are specified, the speaker arrangement unit arranges virtual speakers in a direction in which each virtual sound source included in each of the specified plurality of pairs is located as viewed from the listener, Among the plurality of ranges sandwiched between the arranged virtual speakers, one or more third ranges in which the virtual sound source exists also at positions other than both ends thereof are specified, and the specified third range 5. The sound processing apparatus according to claim 3, wherein the non-arranged virtual speakers are distributed and arranged.   When a plurality of the third ranges are specified, the speaker placement unit calculates an angle centered on the listener between two adjacent virtual speakers in each of the specified third ranges. Then, the process of arranging one unarranged virtual speaker so that the interval between the virtual speakers is even in the third range where the calculated angle is maximum is repeated until there is no unarranged virtual speaker. The speech processing apparatus according to claim 5, wherein:   When calculating the angle centered on the listener between two adjacent virtual speakers in each of the plurality of specified third ranges, the speech synthesizer corresponds to the calculated angle. 7. The speech processing apparatus according to claim 6, wherein weighting is performed according to a position in a range of 3. A plurality of virtual speakers less than the number of the plurality of virtual sound sources arranged around the listener are arranged on a circumference centered on the listener,
Audio signals from each of the plurality of virtual sound sources are distributed to one or more virtual speakers selected for each virtual sound source among the plurality of virtual speakers, and a head related transfer function corresponding to the position of each virtual speaker is used. Synthesize the audio signal distributed to each virtual speaker into left and right channel audio signals;
Including processing,
In the arrangement of the plurality of virtual speakers,
Calculating the angle in the direction along the circumference around the listener, between two virtual sound sources adjacent in the direction along the circumference as seen from the listener;
Of the range of directions along the circumference around the listener, the calculated angle is sandwiched between virtual sound sources that are equal to or greater than a threshold, and the direction of each virtual sound source viewed from the listener Arranging the plurality of virtual speakers in a second range excluding a first range not including the position of
And a voice processing method. On the computer,
A plurality of virtual speakers less than the number of the plurality of virtual sound sources arranged around the listener are arranged on a circumference centered on the listener,
Audio signals from each of the plurality of virtual sound sources are distributed to one or more virtual speakers selected for each virtual sound source among the plurality of virtual speakers, and a head related transfer function corresponding to the position of each virtual speaker is used. Synthesize the audio signal distributed to each virtual speaker into left and right channel audio signals;
Let the process run,
In the arrangement of the plurality of virtual speakers,
Calculating the angle in the direction along the circumference around the listener, between two virtual sound sources adjacent in the direction along the circumference as seen from the listener;
Of the range of directions along the circumference around the listener, the calculated angle is sandwiched between virtual sound sources that are equal to or greater than a threshold, and the direction of each virtual sound source viewed from the listener Arranging the plurality of virtual speakers in a second range excluding a first range not including the position of
A speech processing program characterized by the above.

Images