JP5929455B2 - Audio processing apparatus, audio processing method, and audio processing program - Google Patents

Description

  The present invention relates to a voice processing device, a voice processing method, and a voice processing program.

  In a situation where there are many sound sources around the listener, it is difficult for the listener to distinguish the sound from the desired sound source among those sound sources. In view of this, it is conceived that the listener is put on headphones and the sound corresponding to the sound source selected from the plurality of sound sources is provided to the listener through the headphones.

  For example, a plurality of virtual sound sources are arranged around the listener, the front of the listener's head is directed to a specific sound source, and the listener performs a predetermined operation such as nodding, so that the sound from the specific sound source is There is a technology to choose. There is also a technique for controlling the volume of a virtual sound source arranged in the direction in which the listener is facing to be increased. Further, there is a technique for changing the localization of the sound image according to the orientation of the listener.

Japanese Patent Laid-Open No. 9-90963 JP 2008-92193 A JP 2003-111197 A

  However, in a method in which the listener performs a specific operation such as nodding in order to select a desired sound source from a plurality of virtual sound sources, the operation is complicated, and the listener can perform a natural operation from the desired sound source. There was a problem of not being able to hear the sound.

  In one aspect, an object of the present invention is to provide a voice processing device, a voice processing method, and a voice processing program that enable a listener to recognize a voice corresponding to a desired sound source with natural motion.

  In one proposal, there is provided an audio processing device that controls output of audio signals respectively corresponding to a plurality of virtual sound sources virtually arranged around a listener. This voice processing device includes a state determination unit and an output control unit. The state determination unit determines whether the movement in the listener direction indicating the direction of the listener is in a stationary state. The output control unit determines the volume of the audio signal corresponding to the virtual sound source included in the listening range set so that the listener direction is the center when viewed from the listener, and the sound corresponding to the virtual sound source not included in the listening range. Control to make it relatively larger than the volume of the signal. The output control unit reduces the listening range when it is determined that the stationary state is reached.

Further, in one proposal, a voice processing method is provided in which processing similar to that performed by the voice processing device described above is executed.
Furthermore, in one proposal, a voice processing program that causes a computer to execute the same processing as that of the voice processing device described above is provided.

  According to the first aspect, the listener can distinguish the sound corresponding to the desired sound source with a natural motion.

It is a figure which shows the structural example of the audio | voice processing apparatus which concerns on 1st Embodiment, and its operation example. It is a figure which shows the system configuration example of the audio | voice provision system which concerns on 2nd Embodiment. It is a figure which shows the example of arrangement | positioning of each apparatus in an exhibition hall. It is a figure which shows the hardware structural example of a speech processing unit. It is a figure which shows the hardware structural example of a user terminal. It is a block diagram which shows the structural example of the processing function with which a user terminal and a speech processing unit are provided. It is a figure which shows the example of arrangement | positioning of the sound source in virtual space. It is a figure which shows the example of the information registered into a sound source management table. It is a figure for demonstrating the example of the determination method of a gaze state. It is a figure for demonstrating the listening range. It is a figure for demonstrating the change of the listening range in a two-dimensional virtual space. It is a figure for demonstrating the change of the listening range in a three-dimensional virtual space. It is a figure which shows the 1st example of the method of changing listening range. It is a figure which shows the 2nd example of the method of changing listening range. It is a figure which shows the example of control when a gaze state is canceled before the angle of the listening range becomes the minimum value. It is a figure which shows the example of control when it will be in a gaze state again before the angle of a listening range becomes the maximum value. It is a figure which shows the example of the volume control of each sound source contained in the listening range. It is a figure which shows the example of the information registered into a user management table. It is a flowchart which shows the example of the process sequence of a gaze determination part. It is a flowchart which shows the example of the process sequence of a listening range control part and an audio | voice output process part. It is a flowchart which shows the example of the process sequence of a listening range control part and an audio | voice output process part. It is a figure which shows a mode when the user in a gaze state approaches the exhibit. It is a figure which shows the example of the information registered into an exhibit management table. It is a flowchart which shows the modification of the process sequence of a listening range control part and an audio | voice output process part. It is a figure which shows the structural example of the audio | voice provision system which concerns on 3rd Embodiment. It is a block diagram which shows the example of the processing function of the user terminal and audio | voice processing apparatus in 3rd Embodiment. It is a figure which shows the structural example of the audio | voice provision system which concerns on 4th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example and an operation example of the speech processing apparatus according to the first embodiment.

  The audio processing device 1 shown in FIG. 1 controls output of audio signals respectively corresponding to a plurality of virtual sound sources virtually arranged around the listener 10. The position of each virtual sound source is arbitrarily registered in the sound source position information 2, for example, and held in the storage device of the sound processing device 1. The audio signal corresponding to the virtual sound source is, for example, prepared in a storage device in advance, or collected by a microphone and input to the audio processing device 1. As an example of the latter, there is an audio signal obtained by picking up sound produced by a person who actually exists around the listener 10 using a microphone.

The voice processing device 1 includes a state determination unit 3 and an output control unit 4.
The state determination unit 3 monitors the movement in the listener direction D indicating the direction of the listener 10, and determines whether the movement in the listener direction D has become stationary. For example, a direction sensor is attached to the body of the listener 10, and the state determination unit 3 monitors the movement in the listener direction D based on the detection result by the direction sensor. Note that the listener direction D is preferably the direction in which the face of the listener 10 faces or the direction of the line of sight of the listener 10.

  In addition, the stationary state is a state in which it is determined that the orientation of the listener 10 no longer changes. For example, the state determination unit 3 determines that the stationary state has been reached when the amount of fluctuation in the listener direction D is within a predetermined fluctuation range for a predetermined time.

  The output control unit 4 controls the output volume of an audio signal corresponding to a plurality of virtual sound sources. Here, as an example, it is assumed that the output control unit 4 synthesizes audio signals corresponding to each virtual sound source to generate a synthesized audio signal having a predetermined number of channels. The synthesized voice signal is output to, for example, headphones or earphones worn by the listener 10, and the synthesized voice based on the synthesized voice signal is listened to by the listener 10. Alternatively, synthesized speech based on the synthesized speech signal is output from three or more speakers arranged around the listener 10.

As another example, sound based on a sound signal corresponding to each virtual sound source may be output from a speaker arranged around the listener 10 for each virtual sound source.
The output control unit 4 sets a listening range 30 such that the listener direction D is the center when viewed from the listener 10. Then, the output control unit 4 makes the volume of the audio signal corresponding to the virtual sound source included in the set listening range 30 relatively larger than the volume of the audio signal corresponding to the virtual sound source not included in the listening range 30. To control. By such control, the listener 10 can hear the sound corresponding to the sound sources included in the listening range 30 among the plurality of sound sources with emphasis.

  The output control unit 4 reduces the listening range 30 when the state determination unit 3 determines that the movement in the listener direction D has become stationary. Hereinafter, with reference to the lower side of FIG. 1, an example of the operation before and after it is determined that a stationary state has been described will be described.

  On the lower side of FIG. 1, five virtual sound sources 21 to 25 are arranged around the listener 10. The listening range 30 set by the output control unit 4 is indicated by hatching.

  In a state before it is determined that the state determination unit 3 is in a stationary state, the output control unit 4 sets the listening range 30 as shown in the lower left of FIG. In this state, the listening range 30 includes the virtual sound sources 21 to 25, and the listener 10 can hear sounds corresponding to the virtual sound sources 21 to 25 evenly. For this reason, it is difficult for the listener 10 to distinguish the sound corresponding to each of the virtual sound sources 21 to 25. The state in the lower left of FIG. 1 is a state in which the orientation of the listener 10 is not fixed in a certain direction, and the listener 10 has not specified which sound source corresponding to the sound source among the virtual sound sources 21 to 25 is to be heard. It can be considered a state.

  On the other hand, when it is determined that the state determination unit 3 is in a stationary state, the output control unit 4 reduces the listening range 30 as shown in the lower right of FIG. In this state, the virtual sound source 23 closest to the listener direction D is included in the listening range 30, but the virtual sound sources 21, 22, 24, and 25 are not included in the listening range 30. For this reason, the output control unit 4 makes the volume of the audio signal corresponding to the virtual sound source 23 relatively larger than the volume of the audio signal corresponding to the other virtual sound sources 21, 22, 24, 25. As a result, the listener 10 can easily hear the sound corresponding to the virtual sound source 23.

  With the control as described above, the listener 10 can easily distinguish the sound corresponding to the virtual sound source without consciously performing a special input operation by just standing still in the direction of the desired virtual sound source. Can do. The operation of standing still in the direction of the desired virtual sound source is an operation that the listener 10 unconsciously performs when trying to listen to the sound corresponding to the desired sound source in a situation where there are a plurality of sound sources. For this reason, the listener 10 can listen to the sound corresponding to the desired virtual sound source by natural operation by the above control.

[Second Embodiment]
Next, as a second embodiment, an audio providing system for providing audio information to visitors at an exhibition hall will be described. First, FIG. 2 is a diagram illustrating a system configuration example of a voice providing system according to the second embodiment.

  The audio providing system 100 includes an audio processing device 200 that performs control processing for providing audio information to a user who has entered the exhibition hall. The sound processing apparatus 200 receives sound signals collected by a plurality of microphones. The number of microphones is arbitrary, and as an example in FIG. 2, an audio signal is input to the audio processing device 200 from each of the four microphones 301 a to 301 d. Each microphone picks up sound produced by an explainer explaining the exhibit.

  Note that various methods can be used as a method of transmitting an audio signal from each microphone to the audio processing device 200. For example, an audio signal collected by each microphone is converted into a digital audio signal and then transmitted to the audio processing device 200 by wire or wirelessly. Alternatively, the audio signal collected by each microphone may be input to the audio processing device 200 as an analog signal and digitized in the audio processing device 200.

  In addition, a plurality of access points 110 a to 110 d for transmitting and receiving wireless signals are connected to the audio processing device 200 via a network 120. The network 120 is, for example, a LAN (Local Area Network). In this case, the access points 110a to 110d are wireless LAN access points.

  On the other hand, the user who entered the exhibition hall carries the user terminal 400 and the headphones 500. The user terminal 400 can wirelessly communicate with the access points 110a to 110d. The headphone 500 includes a driver unit (not shown) that reproduces and outputs an analog audio signal output from the user terminal 400.

  The audio processing device 200 synthesizes the audio signals collected by the respective microphones, and transmits the synthesized audio signal to the user terminal 400 through at least one of the access points 110a to 110d. The user terminal 400 converts the audio signal received from the audio processing device 200 into an analog signal, and outputs the converted analog audio signal to the driver unit of the headphone 500.

  The audio processing device 200 has a function of detecting the position of the user terminal 400 in the exhibition hall. As an example in the present embodiment, speech processing apparatus 200 receives signals transmitted from user terminal 400 from access points 110a to 110d, and detects the position of user terminal 400 based on these received signals. For example, the audio processing device 200 receives signals transmitted from the user terminal 400 through the access points 110a to 110d, and triangulation is performed based on the difference in signal reception time or the difference in received radio wave strength at each access point. Is used to detect the position of the user terminal 400. When this method is used, at least three access points used for position detection are installed.

  Furthermore, the headphone 500 is equipped with a direction sensor 510 for detecting the direction in which the user is facing. Hereinafter, the direction detected by the direction sensor 510 is referred to as a “line-of-sight direction”.

  The direction sensor 510 includes, for example, an acceleration sensor, a gyro sensor, and a geomagnetic sensor. The user terminal 400 calculates the user's line-of-sight direction based on the detection result of the direction sensor 510, and transmits the calculated line-of-sight direction to the audio processing device 200 through at least one of the access points 110a to 110d.

  The direction sensor 510 may be provided at a position different from the headphone 500, or may be provided at a position other than the head. However, the purpose of the direction sensor 510 is to detect where the user is looking. For this reason, it is desirable that the direction sensor 510 be provided on the user's head. Further, the direction detected by the direction sensor 510 may be a two-dimensional direction along a horizontal plane or a three-dimensional direction including a vertical direction.

FIG. 3 is a diagram illustrating an arrangement example of each device in the exhibition hall.
In the exhibition hall, for example, exhibits 310a to 310c are exhibited. An exhibitor 302a stands in front of the exhibit 310a, and the explainer 302a explains the exhibit 310a. Similarly, the presenter 302b stands in front of the exhibit 310b, and the presenter 302c stands in front of the exhibit 310c. The explainers 302a to 302c have microphones 301a to 301c, respectively. Then, the sounds emitted by the presenters 302 a to 302 c are collected by the microphones 301 a to 301 c, and the collected sound signals are transmitted to the sound processing apparatus 200.

  The audio processing device 200 synthesizes the received audio signals, and transmits the synthesized audio signals to the user terminal 400 carried by the user 401 through the access points 110a to 110d. The user 401 can directly hear the voices of the presenters 302 a to 302 c by approaching the presenters 302 a to 302 c, but basically listens to the voices of the presenters 302 a to 302 c through the headphones 500.

Note that when a plurality of users 401 enter the exhibition hall, each of the users 401 carries the user terminal 400 and the headphones 500.
By the way, when many exhibits are exhibited in the exhibition hall and there is an explainer for each exhibition, many voices fly in the exhibition hall. The user 401 who has entered the exhibition hall needs to distinguish the voice corresponding to the desired exhibition from these many voices. For example, if the user looks at the desired exhibition, the user 401 corresponds to the exhibition. It is difficult to hear the voice.

  Therefore, the audio processing device 200 acquires the user's 401 gaze direction from the user terminal 400 as needed, and determines whether the movement of the user 401 in the gaze direction is stationary. Hereinafter, a state in which the movement of the user 401 in the line-of-sight direction is determined to be stationary is referred to as a “gaze state”. When the audio processing apparatus 200 determines that the user 401 is in a gaze state, the audio processing apparatus 200 adjusts the synthesis balance of the audio from each microphone so that the audio corresponding to the exhibit existing in the line of sight of the user 401 is emphasized and is easy to hear. After adjustment, the adjusted synthesized speech signal is transmitted to the user terminal 400.

  In addition, when it is determined that the user 401 is in a gaze state, the audio processing device 200 performs control so that the audio corresponding to the exhibit existing in the line of sight of the user 401 is not emphasized suddenly but gently. To do. As a result, the user 401 can hear a desired voice with a natural feeling.

FIG. 4 is a diagram illustrating a hardware configuration example of the audio processing device.
The voice processing apparatus 200 can be realized as a computer as shown in FIG. The entire voice processing apparatus 200 is controlled by a CPU (Central Processing Unit) 201. A RAM (Random Access Memory) 202 and a plurality of peripheral devices are connected to the CPU 201 via a bus 209.

  The RAM 202 is used as a main storage device of the sound processing device 200. The RAM 202 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the CPU 201. The RAM 202 stores various data necessary for processing by the CPU 201.

  Peripheral devices connected to the bus 209 include an HDD (Hard Disk Drive) 203, a graphic interface 204, an input interface 205, an optical drive device 206, a network interface 207, and a communication interface 208.

  The HDD 203 magnetically writes data to and reads data from a built-in magnetic disk. The HDD 203 is used as a secondary storage device of the sound processing device 200. The HDD 203 stores an OS program, application programs, and various data. As the secondary storage device, other types of nonvolatile storage devices such as a flash memory may be used.

  A monitor 204 a is connected to the graphic interface 204. The graphic interface 204 displays an image on the monitor 204a in accordance with a command from the CPU 201. The monitor 204a is a liquid crystal display, for example.

  Input devices such as a keyboard 205a and a mouse 205b are connected to the input interface 205. The input interface 205 transmits an output signal from the input device to the CPU 201.

  The optical drive device 206 reads data recorded on the optical disc 206a using a laser beam or the like. The optical disk 206a is a portable recording medium on which data is recorded so that it can be read by reflection of light. Examples of the optical disk 206a include a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), and a CD-R (Recordable) / RW (Rewritable).

  The network interface 207 transmits / receives data to / from other devices via the network 120. The communication interface 208 receives a digital audio signal collected by each microphone.

FIG. 5 is a diagram illustrating a hardware configuration example of the user terminal.
The user terminal 400 can be realized as an information terminal device as shown in FIG. The entire user terminal 400 is controlled by the CPU 411. A RAM 412 and a plurality of peripheral devices are connected to the CPU 411 via a bus 419.

  The RAM 412 is used as a main storage device of the user terminal 400. The RAM 412 temporarily stores at least part of an OS program and application programs to be executed by the CPU 411. The RAM 412 stores various data necessary for processing by the CPU 411.

  Peripheral devices connected to the bus 419 include a flash memory 413, a display device 414, an input device 415, a wireless interface 416, a communication interface 417, and an audio interface 418.

  The flash memory 413 magnetically writes data to and reads data from a built-in magnetic disk. The flash memory 413 is used as a secondary storage device of the user terminal 400. The flash memory 413 stores an OS program, application programs, and various data. As the secondary storage device, other types of nonvolatile storage devices such as HDDs can also be used.

  The display device 414 includes a liquid crystal display, for example, and displays an image according to a command from the CPU 411. The input device 415 includes, for example, a touch panel installed on the display surface of the display device 414, predetermined operation keys, and the like. A signal corresponding to the operation on the input device 415 is transmitted to the CPU 411.

  The wireless interface 416 performs wireless communication with the access points 110a to 110d. The communication interface 417 receives the detection result from the direction sensor 510. The audio interface 418 converts the digital audio signal transmitted from the CPU 411 into an analog audio signal, amplifies the analog audio signal, and outputs it to the headphones 500.

FIG. 6 is a block diagram illustrating a configuration example of processing functions included in the user terminal and the voice processing device.
The user terminal 400 includes a line-of-sight direction detection unit 421 and a reproduction processing unit 422. Each of these processing blocks is realized, for example, by the CPU 411 (see FIG. 5) of the user terminal 400 executing a predetermined program.

  The gaze direction detection unit 421 calculates the gaze direction ρt of the user 401 based on the detection result by the direction sensor 510. The gaze direction detection unit 421 transmits the calculated gaze direction ρt to the sound processing device 200. The line-of-sight direction ρt is represented by, for example, rotation angles (Rt, Pt, Yt) around the x-axis, y-axis, and z-axis. Alternatively, the line-of-sight direction ρt may be represented as a vector, for example.

  For example, the reproduction processing unit 422 decodes the audio signal received from the audio processing device 200 according to a predetermined encoding method, and supplies the decoded audio signal to the audio interface 418 (see FIG. 5). Further, the reproduction processing unit 422 may perform, for example, a process for giving a pseudo 3D effect to the received audio signal.

  On the other hand, the audio processing device 200 includes a user position detection unit 211, an audio input unit 212, a time measurement unit 213, a gaze determination unit 214, a listening range control unit 215, and an audio output processing unit 216. The processing of each processing block is realized, for example, by the CPU 201 (see FIG. 4) of the sound processing device 200 executing a predetermined program.

  In addition, a sound source management table 220 and a user management table 230 are stored in the storage device of the sound processing device 200. In the sound source management table 220, information related to audio corresponding to each exhibit is registered. In addition, information regarding each user 401 is registered in the user management table 230. The sound source management table 220 and the user management table 230 are, for example, developed in the RAM 202 (see FIG. 4) of the sound processing device 200 and read / written from the processing block of the sound processing device.

  The user position detection unit 211 receives signals transmitted from the user terminal 400 through the access points 110a to 110d, and detects the position of the user terminal 400 based on these received signals. The user position detection unit 211 registers the detected position of the user terminal 400 as the user position Qt in the user management table 230 corresponding to the detection target user as needed.

  The voice input unit 212 receives a voice signal picked up by microphones respectively provided by a plurality of explainers as a sound source, and supplies the received voice signal to the voice output processing unit 216.

  In the sound source management table 220, information on each sound source received by the voice input unit 212 is registered. As will be described later, each sound source is arranged at an arbitrary position in the virtual space corresponding to the exhibition hall, and the sound source management table 220 registers position information of each sound source in the virtual space.

The timer unit 213 supplies the current time to the gaze determination unit 214 and the listening range control unit 215. The time from the time measuring unit 213 is used when various elapsed times are obtained.
The gaze determination unit 214 determines whether the user 401 is in a gaze state based on the gaze direction ρt received from the user terminal 400. As described above, the gaze state is a state in which it is determined that the movement in the line-of-sight direction ρt is stationary. The gaze determination unit 214 notifies the listening range control unit 215 of the determination result as to whether or not the user is in the gaze state. In addition, the gaze determination unit 214 registers the line-of-sight direction ρt received from the user terminal 400 in the user management table 230 corresponding to the detection target user 401 as needed.

  The listening range control unit 215 controls the size of the listening range set for each user 401 in the virtual space. The listening range is a range centered on the viewing direction ρt as viewed from the user 401, and the listening range is determined by an angle formed with the viewing direction ρt, as will be described later. The volume of the audio signal corresponding to the sound source included in the listening range among the sound sources arranged in the virtual space is controlled to be larger than the volume of the audio signal corresponding to the sound source not included in the listening range. . In the present embodiment, as an example, only the sound from the sound source included in the listening range is provided to the user 401 among the sound sources arranged in the virtual space, and the sound from the sound source not included in the listening range is provided to the user 401. It is controlled so that it cannot be heard.

  When the gaze determination unit 214 determines that the user 401 is in the gaze state, the listening range control unit 215 gradually narrows the listening range until a predetermined minimum size is reached. By such control, the listening range control unit 215 gradually makes it easier to hear the sound from the sound source arranged in the direction in which the user 401 is facing in the sound transmitted to the user terminal 400.

  The audio output processing unit 216 selects and synthesizes audio signals for sound sources included in the listening range set by the listening range control unit 215 from the audio signals input from the audio input unit 212, and outputs the left and right channels one by one. The synthesized voice signal is generated. The audio output processing unit 216 transmits the generated synthesized audio signal to the user terminal 400.

Next, FIG. 7 is a diagram illustrating an arrangement example of sound sources in the virtual space.
The virtual space 320 is a space that represents an exhibition hall where the user 401 and the exhibits 310a to 310j exist by a two-dimensional or three-dimensional coordinate system. FIG. 7 shows an example in which the virtual space 320 is represented by a three-dimensional coordinate system using the x axis, the y axis, and the z axis. The user position Qt is represented by coordinates (Xt, Yt, Zt). Further, the line-of-sight direction ρt of the user 401 is expressed as (Rt, Pt, Yt) using, for example, a rotation angle around each axis.

  The sound sources P1 to P10 correspond to audio signals obtained by collecting the voices of the explainers explaining the exhibits 310a to 310j. The sound sources P <b> 1 to P <b> 10 are arranged at the positions of the exhibits 310 a to 310 j in the virtual space 320. For example, the sound source P1 corresponds to the audio signal of the explainer explaining the exhibit 310a, and the sound source P2 corresponds to the audio signal of the explainer explaining the exhibit 310b. The sound source P1 is virtually arranged at the position of the exhibit 310a, and the sound source P2 is virtually arranged at the position of the exhibit 310b. The position of the sound source is arranged at a position representing the corresponding exhibit, such as the center of the corresponding exhibit.

  The positions of the sound sources P1 to P10 in the virtual space 320 are set in the sound source management table 220. The administrator of the sound processing device 200 can arbitrarily set the position of each sound source by an input operation to the sound processing device 200.

In FIG. 7, coordinates indicating the position of the sound source Pn (n is an integer of 1 or more) in the virtual space 320 are expressed as (Xn, Yn, Zn).
In this embodiment, the sound emitted by the exhibitor of the exhibit is used as the sound source. However, the sound source may be, for example, reproduced sound based on an audio signal stored in the storage device in advance. In this case, for example, an audio signal used as a sound source is stored in the HDD 203 of the audio processing apparatus 200, and the audio input unit 212 supplies the audio signal read from the HDD 203 to the audio output processing unit 216.

FIG. 8 is a diagram illustrating an example of information registered in the sound source management table.
In the sound source management table 220, for each sound source arranged in the virtual space, a sound source ID for uniquely identifying each sound source and a sound source coordinate indicating a position where the sound source is arranged in the virtual space are registered in association with each other. Has been. The administrator of the sound processing apparatus 200 can perform operations such as adding new sound source information to the sound source management table 220, changing sound source coordinates, and deleting sound source information from the sound source management table 220. .

  In FIG. 8, for example, an exhibit ID for uniquely identifying an exhibit associated with the sound source indicated by the sound source ID is registered. For example, the administrator of the sound processing apparatus 200 can associate a plurality of sound sources with one exhibit, or can associate one sound source with a plurality of exhibits.

Next, FIG. 9 is a diagram for explaining an example of a gaze state determination method.
As described above, the gaze state is a state in which the movement of the user 401 in the line-of-sight direction ρt is determined to be stationary. However, even when the user 401 is gazing at a specific position, in practice, the movement of the user 401 is rarely completely stopped. In view of this point, the gaze determination unit 214 of the audio processing device 200 determines that the user 401 is in a gaze state when the variation amount of the value of the line-of-sight direction ρt is within a certain threshold width Wth for a predetermined time. judge.

  In the example of FIG. 9, the amount of change in the value of the line-of-sight direction ρt (angle Rt in FIG. 9) falls within the threshold width Wth during the period from time t1 to time t2 when a predetermined determination time Ta has elapsed. Yes. In this case, the gaze determination unit 214 determines that the user 401 is in a gaze state at time t2. Further, when the amount of change in the line-of-sight direction ρt deviates from the threshold width Wth at the subsequent time t3, the gaze determination unit 214 determines that the gaze state has been eliminated.

  It should be noted that the determination as to whether or not the user 401 is in the gaze state actually has all the fluctuation amounts of the axial direction values (that is, Rt, Pt, Yt) indicating the line-of-sight direction ρt within the threshold width Wth. Done depending on what you are doing.

Next, FIG. 10 is a diagram for explaining the listening range. FIG. 10 shows an example of a listening range in a two-dimensional virtual space as an example.
The listening range is set around the user position Qt. When the virtual space is defined by two-dimensional coordinates as shown in FIG. 10, the listening range is defined by an angle θt that the boundary of the listening range makes with the visual line direction ρt of the user 401 on the horizontal plane (xy plane). In the case of a two-dimensional space, the boundaries referred to here are two straight lines extending radially from the user position Qt. The range between the one boundary line and the line-of-sight direction ρt and the range between the other boundary line and the line-of-sight direction ρt are the listening range.

  In the example of FIG. 10A, the listening range refers to a range of θ1 degrees with respect to the clockwise direction and the counterclockwise direction from the line-of-sight direction ρt with the user position Qt as the center. The maximum value of the listening range angle θt is 180 degrees, and in this embodiment, the initial value of the listening range angle θt is 180 degrees, which is the maximum value. In addition, the minimum value of the angle θt of the listening range is set to a predetermined angle greater than 0 degrees.

  Here, as shown in FIG. 10, it is assumed that sound sources P <b> 1 to P <b> 4 and P <b> 9 are arranged at predetermined positions around the user 401 in the virtual space. The audio processing device 200 synthesizes audio signals of sound sources included in the listening range when viewed from the user position Qt, and transmits the synthesized audio signal to the user terminal 400.

  In the example of FIG. 10A, since the sound sources P1 to P4 and P9 are included in the listening range, the sound processing device 200 synthesizes the sound signals of the sound sources P1 to P4 and P9 to generate a combined sound signal. Is transmitted to the user terminal 400. On the other hand, in the example of FIG. 10B, the angle θt of the listening range is set to θ2, which is smaller than θ1, and the listening range is narrower than in the case of FIG. At this time, the sound sources P1 to P3 are included in the listening range, but the sound sources P4 and P9 are not included in the listening range. In this state, the audio processing device 200 synthesizes the audio signals of the sound sources P <b> 1 to P <b> 3 and transmits the synthesized audio signal to the user terminal 400. Therefore, the user 401 cannot hear the sounds of the sound sources P4 and P9.

  By narrowing the angle θt of the listening range, the user 401 can hear only the sound of the sound source arranged near the line-of-sight direction ρt. As a result, the user 401 can easily hear the sound emitted from the direction in which the user 401 is facing.

  Note that the sound processing device 200 localizes the sound image of each sound source included in the listening range to a corresponding position in the left-right direction with respect to the line-of-sight direction ρt based on the relative relationship between the user position Qt and the position of each sound source. Let In the example of FIG. 10, the audio processing device 200 increases the volume of the left channel with respect to the audio signal of the sound source P1, and increases the volume of the right channel with respect to the audio signal of the sound source P3. To do.

  In the present embodiment, audio processing apparatus 200 does not synthesize an audio signal corresponding to a sound source not included in the listening range into a synthesized audio signal. However, the sound processing apparatus 200 synthesizes a sound signal corresponding to a sound source not included in the listening range with a synthesized sound signal with a volume lower than that of the sound signal corresponding to the sound source included in the listening range. Also good.

  Next, the relationship between the movement in the line-of-sight direction ρt and the angle of the listening range will be described with reference to FIGS. 11 and 12. First, FIG. 11 is a diagram for explaining a change in the listening range in a two-dimensional virtual space.

  The audio processing device 200 sets the angle θt of the listening range to the maximum value θmax (= 180 degrees) when the movement of the user 401 in the line-of-sight direction ρt is not stationary (non-gaze state). In this state, the user 401 can hear the sound of all the sound sources arranged around. When the sound processing device 200 determines that the movement in the line-of-sight direction ρt is a “gaze state” in which the movement is stationary, the sound processing apparatus 200 narrows the angle θt of the listening range. At this time, the audio processing device 200 does not immediately change the angle θt of the listening range to the minimum value θmin, but gradually narrows it as time passes. In the example of FIG. 11, after transitioning to the gaze state, the angle θt of the listening range is gradually narrowed from θmax to θ1, θ2, θmin.

  As described above, when the user 401 is in a gaze state, the angle θt of the listening range is narrowed, and the number of sound sources that can be heard by the user 401 among the sound sources arranged around the user 401 decreases. Eventually, the user 401 can hear only the sound of the sound source included in the narrow range only around the direction in which the user 401 is facing, that is, the listening range in which the angle θmin is set. Further, by gradually narrowing the listening range, the user 401 can listen to the sound emitted from the direction in which he / she is facing with a natural feeling.

FIG. 12 is a diagram for explaining a change in the listening range in the three-dimensional virtual space.
When the virtual space is defined by a three-dimensional coordinate system, the listening range is defined by an angle θt that makes the boundary of the listening range the visual line direction ρt in the horizontal direction and an angle φt that makes the visual line direction ρt in the vertical direction. Is done. When the movement in the line-of-sight direction ρt is not stationary (non-gaze state), the audio processing device 200 sets the horizontal angle θt of the listening range to the maximum value θmax (= 180 degrees) and the vertical range of the listening range. The direction angle φt is set to the maximum value φmax (= 180 degrees). In this state, the user 401 can hear the sound of all the sound sources arranged around.

  When the sound processing apparatus 200 determines that the movement in the line-of-sight direction ρt is in a “gaze state” where the movement is stationary, both the angles θt and φt of the listening range are gradually narrowed. In the example of FIG. 12, after transitioning to the gaze state, the horizontal angle θt of the listening range is gradually narrowed from θmax to θ1, θ2, θmin, and the vertical angle φt of the listening range is changed from φmax. It is gradually narrowed like φ1, φ2, φmin.

Next, an example of a method for changing the listening range will be described with reference to FIGS. 13 and 14. FIG. 13 is a diagram illustrating a first example of a method for changing the listening range.
In the example of FIG. 13, during the period from time td to time t <b> 0 when the determination time Ta has elapsed, the fluctuation amount of the user 401 in the line-of-sight direction ρt falls within the predetermined threshold width Wth. In this case, the gaze determination unit 214 determines that the user 401 is in a gaze state at time t0. The listening range control unit 215 starts the time t0 as a starting point and decreases the listening range angle θt according to the following equation (1).
θt = (θmax−θmin) / {1 + EXP [((ts−t0) − (Tf / 2)) × a]} + θmin (1)
In Equation (1), ts indicates the current time. Further, as shown in FIG. 13, Tf indicates a time until the angle θt of the listening range reaches the minimum value θmin from the maximum value θmax, and can be arbitrarily set by, for example, the administrator of the sound processing apparatus 200. Further, a is a constant that can be arbitrarily set, and is set to a = 12 / Tf, for example.

In the example of FIG. 13, the gaze determination unit 214 determines that the gaze state has been resolved at time t0 ′ after the user 401 has entered the gaze state. The listening range control unit 215 starts the time t0 ′ and increases the listening range angle θt according to the following equation (2).
θt = (θmax−θmin) / {1 + EXP [− ((ts−t0 ′) − (Tf ′ / 2)) × a]} + θmin (2)
In Expression (2), Tf ′ indicates the time until the angle θt of the listening range reaches the maximum value θmax from the minimum value θmin, as shown in FIG. Tf ′ can be arbitrarily set by the administrator of the speech processing apparatus 200, and is set to the same value as Tf, for example.

FIG. 14 is a diagram illustrating a second example of a method for changing the listening range.
In the example of FIG. 14 as well, in the period from the time td to the time t0 when the determination time Ta has elapsed, the variation amount of the user 401 in the line-of-sight direction ρt falls within the predetermined threshold width Wth, as in the example of FIG. . The gaze determination unit 214 determines that the user 401 is in a gaze state at time t0. The listening range control unit 215 starts the time t0 as a starting point and decreases the listening range angle θt according to the following equation (3).
θt = {b ^{[(ts−t0) −Tf / 2]} } + ^θmin (3)
In equation (3), b is a constant that can be set arbitrarily.

When the gaze determination unit 214 determines that the gaze state has been resolved at time t0 ′, the listening range control unit 215 increases the angle θt of the listening range according to the following equation (4), starting from time t0 ′. Let
θt = {c × log [(ts−t0 ′) + d] + θmin (4)
In Equation (4), c and d are constants that can be arbitrarily set.

  13 and 14 described the listening range angle θt, but the listening range angle φt may also be expressed by the above formulas (1) and (2) or the formulas (3) and (4). ) Can be controlled according to.

FIG. 15 is a diagram illustrating a control example when the gaze state is canceled before the angle of the listening range reaches the minimum value.
When the gaze determination unit 214 determines that the gaze state is entered at time t0, the listening range control unit 215 gradually decreases the listening range angle θt. However, it is assumed that the gaze determination unit 214 determines that the gaze state has been resolved at time t11 when the time Tf has not elapsed since time t0, that is, the angle θt of the listening range has not decreased to the minimum value θmin. In this case, for example, the listening range control unit 215 gradually increases the angle θt of the listening range until the maximum value θmax is reached.

FIG. 16 is a diagram illustrating a control example in a case where the user is in a gaze state again before the angle of the listening range reaches the maximum value.
When the gaze determination unit 214 determines that the gaze state has been eliminated at time t0 ′, the listening range control unit 215 gradually increases the angle θt of the listening range, for example. However, suppose that the gaze determination unit 214 determines that the gaze state is again in the gaze state at the time t12 when the time Tf ′ has not elapsed since the time t0 ′, that is, the angle θt of the listening range has not reached the minimum value θmax. In this case, the listening range control unit 215 gradually decreases the listening range angle θt until it reaches the minimum value θmin.

  In the example of FIGS. 13 to 16 described above, the listening range control unit 215 gradually widens the listening range with time when the gaze state is resolved. However, the listening range control unit 215 may immediately set the listening range to the maximum value θmax, for example, when the gaze state is resolved. Alternatively, the listening range control unit 215 may adjust the speed at which the angle θt of the listening range is increased according to the degree of change in the line-of-sight direction ρt when the gaze state is resolved.

FIG. 17 is a diagram illustrating an example of volume control of each sound source included in the listening range.
When the audio output processing unit 216 of the audio processing device 200 synthesizes audio signals corresponding to a plurality of sound sources included in the listening range, for example, a sound source arranged at a position close to the center of the listening range as viewed from the user 401 As a result, the volume of the corresponding audio signal may be increased. As a result, the user 401 can hear a louder sound as the sound source is closer to the direction in which the user 401 is facing, and the sound becomes natural.

As a volume control method, for example, the following method can be used.
The range of the angle θt of the listening range is divided into a plurality of ranges in advance. Then, for each divided range, a gain value for multiplying the audio signal corresponding to the sound source existing in the range is set. The gain value is set larger as it is closer to the center of the listening range.

  In the example of FIG. 17, the range from the center of the listening range (angle is 0 degrees) to the angle θmax (= 180 degrees) is divided into five ranges. A gain “1” is set in the range from the center of the listening range to the angle θmin. The gain “0.8” is set in the range of the next angle θ1 from the angle θmin of the listening range, the gain “0.6” is set in the range of the next angle θ2, and the next range of θ3. A gain “0.4” is set, and a gain “0.2” is set in the range up to the next angle θ4, that is, the range up to the angle θmax.

  When generating the synthesized speech signal, the sound output processing unit 216 determines in which of the above-described divided ranges each sound source exists, and sets the gain corresponding to the determined range to the level of the sound signal of the sound source. To synthesize the audio signal after gain adjustment.

  In the example of FIG. 17, the gain is fixedly set with respect to the angle of the listening range. However, for example, the gain that is set according to the change in the size of the listening range may change. As an example of this case, the following control method can be used.

  The range from the center of the listening range (the angle is 0 degree) to the angle θt currently set by the listening range control unit 215 is divided according to the ratio, and a gain is set for each divided range. However, since the position of the sound source desired by the user 401 does not necessarily completely overlap with the line-of-sight direction ρt of the user 401, it is desirable that the range in which the gain “1” is set be maintained at a constant size.

  Therefore, for example, the gain “1” is set in a fixed range from 0 degree to the angle θmin in the listening range. In the range from 0 degree to the angle θt in the listening range, the gain “0.8” is set in the range from the angle θmin to the next θ1 / θt, and the gain “0” is set in the next θ2 / θt range. .6 ”, a gain“ 0.4 ”is set in the next range of θ3 / θt, and a gain“ 0.2 ”is set in the next range of θ4 / θt. When the gain is assigned in this way, the volume of the sound source arranged in the same direction as viewed from the user 401 (except for the sound source arranged in the range of θmin from the center) gradually increases as the listening range becomes narrower. Becomes smaller. Therefore, the sound of a desired sound source is emphasized with a natural feeling for the user 401.

  Note that the gain control method for the sound source as described above may be used only during a period in which the gaze determination unit 214 determines that the gaze state is in the gaze state, for example. Specifically, in a period in which it is determined that the user is not in the gaze state, the audio output processing unit 216 outputs audio signals corresponding to all sound sources included in the listening range with the same volume ratio (that is, gain “1” for all). ”And combine). In this state, the user 401 can hear the sound corresponding to all the sound sources included in the listening range evenly. However, since the user 401 is considered to have not yet specified the sound source to be heard, it is not particularly unnatural.

  Then, when it is determined that the state of gaze is reached, the audio output processing unit 216 increases the volume of the audio signal corresponding to the sound source included in the listening range as described above, as the sound source is closer to the center of the listening range. To control. When the user 401 is in a gaze state, the user 401 can easily hear the sound corresponding to the sound source that is close to the line-of-sight direction ρt. Further, by further reducing the listening range, the user 401 can hear the sound corresponding to the sound source close to the line-of-sight direction ρt more clearly.

Next, FIG. 18 is a diagram illustrating an example of information registered in the user management table.
Records are registered in the user management table 230 for each user 401. In each record, user coordinates, line-of-sight direction, rest time, gaze flag, effective sound source ID, listening range angle, and non-gaze time are registered in association with the user ID for identifying the user 401.

The user coordinates are coordinates on the virtual space indicating the above-described user position Qt, and are updated by the user position detection unit 211 as needed.
The line-of-sight direction corresponds to the above-described line-of-sight direction ρt, and here is represented by a rotation angle (Rt, Pt, Yt) around each axis as an example. The gaze direction is updated by the gaze determination unit 214 as needed.

  The stationary time is a time during which the gaze determination unit 214 determines that the line-of-sight direction ρt is within the threshold width Wth (see FIG. 9). For example, the stationary time is an elapsed time from time t1 in FIG. 9, and is an elapsed time from time td in FIGS. The stationary time is set by the gaze determination unit 214. In the present embodiment, it is assumed that the stationary time is registered in units of 1 second.

  Note that the stationary time is referred to by the listening range control unit 215 when the listening range angles θt and φt are decreased. Here, the elapsed time after determining that the gaze determination unit 214 is in the gaze state ((ts−t0) in the expressions (1) and (3)) is determined from the stationary time to the determination time Ta (FIG. 9, FIG. 9). 12 to 14) is subtracted.

  The gaze flag is flag information indicating whether or not the user is in the gaze state, and is set to “1” when the gaze state is set, and is set to “0” when the gaze state is not set. The gaze flag is set by the gaze determination unit 214.

  The effective sound source ID is a sound source ID indicating a sound source included in the listening range, and is set by the listening range control unit 215. When no sound source is included in the listening range, “0” is set as the effective sound source ID.

The listening range angles are the above-described listening range angles θt and φt, and are set by the listening range control unit 215.
The non-gaze time indicates an elapsed time after the gaze state is resolved. For example, the non-gaze time is an elapsed time from time t′0 in FIGS. The non-gaze time is set by the gaze determination unit 214 and is referred to by the listening range control unit 215 when increasing the angles θt and φt of the listening range.

  Next, FIG. 19 is a flowchart illustrating an example of a processing procedure of the gaze determination unit. The process in FIG. 19 is executed for each user 401. In addition, the gaze determination unit 214 sets a stationary time “0”, a non-gaze time “0”, and a gaze flag “0” as initial values in the user management table 230 at the start of the processing of FIG.

  [Step S11] The gaze determination unit 214 captures the detection result of the line-of-sight direction ρt received from the user terminal 400 for one second. Note that the detection result of the line-of-sight direction ρt is transmitted from the user terminal 400 a plurality of times per second. The gaze determination unit 214 registers the received value in the line of sight direction field of the user management table 230 every time it receives the detection result of the line of sight direction ρt.

  [Step S12] The gaze determination unit 214 determines whether all of the fluctuation amounts for each axis of the captured gaze direction ρt for one second are within the threshold width Wth. The gaze determination unit 214 executes the process of step S13 when the variation amount is within the threshold width Wth. On the other hand, the gaze determination unit 214 executes the process of step S21 when at least one of the fluctuation amounts for each axis does not fall within the threshold width Wth.

[Step S13] The gaze determination unit 214 increments the value of the still time in the user management table 230 by “1”.
[Step S14] The gaze determination unit 214 captures the detection result of the line-of-sight direction ρt received from the user terminal 400 for one second. As in step S11, the gaze determination unit 214 registers the received value in the line of sight direction field of the user management table 230 every time it receives the detection result of the line of sight direction ρt.

  [Step S15] The gaze determination unit 214 determines whether or not the amount of variation for each axis in the line-of-sight direction ρt in the period from the determination of “Yes” in Step S12 to the present time is all within the threshold width Wth. The gaze determination unit 214 executes the process of step S16 when the variation amount is within the threshold width Wth. On the other hand, the gaze determination unit 214 executes the process of step S20 when at least one of the fluctuation amounts for each axis does not fall within the threshold width Wth.

[Step S16] The gaze determination unit 214 increments the value of the stationary time in the user management table 230 by “1”.
[Step S17] The gaze determination unit 214 determines whether or not the elapsed time after determining “Yes” in Step S12 has reached the determination time Ta. The elapsed time mentioned here is the number of seconds registered in the column of the still time in the user management table 230. When the elapsed time reaches the determination time Ta, the gaze determination unit 214 executes the process of step S19. On the other hand, when the elapsed time has not reached the determination time Ta, the gaze determination unit 214 executes the process of step S18.

  [Step S18] The state determined as “No” in Step S17 is a state where the user 401 is not in a gaze state (non-gaze state). In this case, the gaze determination unit 214 increments the value of the non-gaze time in the user management table 230 by “1”. Thereafter, the process of step S14 is executed.

  [Step S19] The state determined as “Yes” in Step S17 is a state where it is determined that the user 401 is in a gaze state. In this case, the gaze determination unit 214 updates the value of the gaze flag in the user management table 230 to “1” and resets the value of the non-gaze time to “0”. Thereafter, the process of step S14 is executed.

  [Step S20] The state determined as “No” in Step S15 is a state in which the movement of the direction of the user 401 has increased. In this case, the gaze determination unit 214 resets the value of the stationary time in the user management table 230 to “0”. In addition, when the value of the gaze flag in the user management table 230 is “1”, the gaze determination unit 214 updates this value to “0”. Thereafter, the process of step S21 is executed.

[Step S21] The gaze determination unit 214 increments the value of the non-gaze time in the user management table 230 by “1”. Thereafter, the process of step S11 is executed.
20 and 21 are flowcharts illustrating examples of processing procedures of the listening range control unit and the audio output processing unit. The processing in FIGS. 20 and 21 is executed for each user 401. Further, for example, control is performed so that the process of step S31 in FIG. 20 is executed at a cycle of one audio frame.

  [Step S31] The listening range control unit 215 checks the value of the gaze flag in the user management table 230. When the value of the gaze flag is “1”, the listening range control unit 215 executes the process of step S32. On the other hand, when the value of the gaze flag is “0”, the listening range control unit 215 executes the process of step S51 of FIG.

  [Step S <b> 32] The listening range control unit 215 calculates an elapsed time after the gaze determination unit 214 determines that the gaze state is entered based on the value registered in the static time column in the user management table 230. Then, the listening range control unit 215 determines whether the elapsed time is less than the time Tf. When the elapsed time is less than the time Tf, the listening range control unit 215 executes the process of step S33. On the other hand, when the elapsed time is equal to or longer than the time Tf, the listening range control unit 215 executes the process of step S34.

  [Step S33] The state determined as “Yes” in Step S32 is a state in which the angles θt and φt of the listening range are being gradually decreased. In this case, the listening range control unit 215 calculates the angles θt and φt of the listening range according to the above formula (1) or formula (3) according to the elapsed time calculated in step S32. The listening range control unit 215 registers the calculated angles θt and φt in the listening range angle column in the user management table 230.

  [Step S34] The listening range control unit 215 determines the angles θt and φt of the listening range as predetermined minimum values, and registers the determined values in the listening range angle column of the user management table 230.

  [Step S35] The listening range control unit 215 listens based on the values in the fields of the user coordinates, the line-of-sight direction, and the listening range angle in the user management table 230 and the position information of each sound source registered in the sound source management table 220. Check the sound sources included in the listening range set by the range angle value.

  [Step S36] The listening range control unit 215 determines whether one or more sound sources are included in the listening range. If one or more sound sources are included in the listening range, the listening range control unit 215 executes the process of step S38. On the other hand, the listening range control unit 215 executes the process of step S37 when no sound source is included in the listening range.

  [Step S37] The listening range control unit 215 corrects the angles θt and φt registered in the listening range angle column in the user management table 230 by increasing them by one step. Thereafter, the process of step S35 is executed.

[Step S38] The listening range control unit 215 registers the sound source ID of the sound source included in the listening range in the column of the effective sound source ID in the user management table 230.
[Step S39] The audio output processing unit 216 uses the user coordinates, line-of-sight direction, and listening range angle fields in the user management table 230, and the position information of each sound source registered in the sound source management table 220 based on the user information. For each sound source registered in the column of effective sound source ID in the management table 230, a gain at the time of synthesizing the synthesized sound signal is determined.

  For example, the audio output processing unit 216 sets the gain of the corresponding audio signal to be larger as the sound source is closer to the center of the listening range in accordance with the processing procedure described in FIG. In addition, the audio output processing unit 216 depends on the angle formed by the line connecting the user position Qt and the position of the sound source and the line-of-sight direction ρt, and whether the sound source is arranged on the left or right with respect to the center of the listening range. Then, the volume balance in each of the left channel and the right channel is adjusted, and the sound image of the sound source is localized at any position in the left-right direction.

  [Step S <b> 40] The audio output processing unit 216 generates a synthesized audio signal by applying the gain determined in step S <b> 39 and transmits the synthesized audio signal to the user terminal 400. Thereafter, the process returns to step S31.

  [Step S51] The listening range control unit 215 calculates an elapsed time after the gaze determination unit 214 determines that the gaze state has been resolved based on the value registered in the non-gaze time column in the user management table 230. . Then, the listening range control unit 215 determines whether the elapsed time is less than the time Tf ′. When the elapsed time is less than the time Tf ′, the listening range control unit 215 executes the process of step S52. On the other hand, when the elapsed time is equal to or longer than the time Tf ′, the listening range control unit 215 executes the process of step S53.

  [Step S52] The state determined as “Yes” in Step S51 is a state in which the angles θt and φt of the listening range are being gradually increased. In this case, the listening range control unit 215 calculates the angles θt and φt of the listening range according to the above formula (2) or formula (4) according to the elapsed time calculated in step S51. The listening range control unit 215 registers the calculated angles θt and φt in the listening range angle column in the user management table 230.

  [Step S <b> 53] The listening range control unit 215 determines the listening range angles θt and φt to predetermined maximum values, and registers the determined values in the listening range angle column of the user management table 230.

Note that the listening range control unit 215 may unconditionally execute the process of step S53 without performing the determination of step S51.
[Step S54] Based on the values in the fields of the user coordinates, line-of-sight direction, and listening range angle in the user management table 230, and the position information of each sound source registered in the sound source management table 220, the listening range angle value is set. Check the sound sources included in the listening range. The listening range control unit 215 registers the sound source ID of the sound source included in the listening range in the column of the effective sound source ID in the user management table 230. Thereafter, the process of step S39 is executed.

  According to FIGS. 20 and 21 described above, when the gaze determination unit 214 determines that the gaze state is reached, the listening range is gradually reduced, and the sound corresponding to the sound source near the center of the listening range is gradually increased. It is emphasized and can be heard by the user 401. Therefore, the user 401 can easily recognize the sound corresponding to the sound source arranged in the direction in which the user 401 is facing.

  Further, in steps S36 and S37 of FIG. 20, the listening range control unit 215 adjusts the angle of the listening range so that the sound source is always present in the listening range when in the gaze state. Here, FIG. 22 is a diagram illustrating a state when a user in a gaze state approaches the exhibit.

  In state 1 in FIG. 22, the user 401 is in a state of gazing while facing the exhibit 310a. The distance between the user position Qt and the exhibit 310a is d1, and the angle θt of the listening range is the angle θ1. The listening range includes the sound source P1 corresponding to the exhibit 310a. For this reason, the user 401 can hear the sound corresponding to the sound source P <b> 1 through the headphones 500.

  Here, as shown in state 2, it is assumed that the user 401 approaches the exhibit 310 a while the gaze state is maintained. At this time, it is assumed that the distance between the user position Qt and the exhibit 310a is d2. When the angle θt of the listening range remains the angle θ1, the sound source P1 corresponding to the exhibit 310a may not be included in the listening range as in the state 2. In this case, the user 401 cannot hear the sound corresponding to the sound source P1.

  According to steps S36 and S37 in FIG. 20 described above, when the listening range is reduced, if no sound source is included in the listening range, the listening range is expanded until at least one sound source is included. Is done. Such a process can avoid a situation in which the user 401 cannot hear the corresponding sound when the user 401 approaches the exhibit 310a as in the state 2 of FIG.

  As another method, the listening range control unit 215 may detect the distance between the user position Qt and the exhibit and correct the angle so that the angle of the listening range increases as the distance decreases. With this method, for example, as shown in state 3 of FIG. 20, even when the distance between the user position Qt and the exhibit 310a approaches d2, the possibility that the sound source P1 is included in the listening range is increased.

  A method of correcting the angle of the listening range according to the distance between the user position Qt and the exhibit can be realized by the following control. The storage device of the sound processing device 200 stores an exhibit management table as shown in FIG.

FIG. 23 is a diagram illustrating an example of information registered in the exhibit management table.
In the exhibit management table 240, a record is registered for each exhibit. In each record, information indicating an area where the exhibit is arranged is registered with respect to the exhibit ID for identifying the exhibit. . In the example of FIG. 23, the number of vertices and vertex coordinates are registered as information indicating the arrangement area of the exhibit. The number of vertices indicates the number of vertices existing at the peripheral edge of the exhibit. The vertex coordinates indicate the coordinates of each vertex in the virtual space.

  FIG. 24 is a flowchart illustrating a modification of the processing procedure of the listening range control unit and the audio output processing unit. In FIG. 24, processing steps in which the same processing as that in FIG. 20 is executed are denoted by the same reference numerals, and description of these processing steps is omitted.

In the process shown in FIG. 24, steps S61 and S62 are executed instead of steps S35 to S37 in FIG.
[Step S61] The listening range control unit 215 calculates the distance from the exhibit associated with the sound source included in the listening range. Specifically, the listening range control unit 215 refers to the sound source management table 220 and identifies exhibits that correspond to the sound sources included in the listening range. The listening range control unit 215 reads the number of vertices and vertex coordinates from the exhibit management table 240 for each identified exhibit, and calculates the distance between each vertex and the user position Qt. The listening range control unit 215 sets the smallest calculation result among the calculation results of the distances between all the vertices of all the identified exhibits and the user position Qt as the distance to the exhibit.

  [Step S62] The listening range control unit 215 corrects the angles θt and φt registered in the listening range angle of the user management table 230 according to the calculated distance from the exhibit. For example, the listening range control unit 215 performs correction by multiplying the registered angles θt and φt by a correction coefficient corresponding to the distance from the exhibit. For example, the listening range control unit 215 sets the correction coefficient to “1” when the distance to the exhibit is equal to or greater than a predetermined lower limit. When the distance to the exhibit is less than the lower limit value, the listening range control unit 215 sets the correction coefficient smaller than “1” as the distance becomes shorter.

  As a result of the processes in steps S61 and S62 described above, the listening range is expanded as the user 401 approaches the exhibit, and thus the user 401 cannot hear the sound corresponding to the exhibit at the destination of the user 401. Less likely to occur.

  In the above-described processing of FIG. 24, the angle of the listening range is corrected according to the distance between the user position Qt and the exhibit, but as another example, the listening according to the distance between the user position Qt and the sound source position. The angle of the range may be corrected.

[Third Embodiment]
FIG. 25 is a diagram illustrating a configuration example of a voice providing system according to the third embodiment. 25, components corresponding to those in FIG. 3 are denoted by the same reference numerals, and description thereof is omitted. Hereinafter, differences between the third embodiment and the second embodiment will be described.

  In the audio providing system according to the third embodiment, the user 401 listens to audio output from a plurality of speakers installed in the exhibition hall instead of listening to audio from the headphones 500. Therefore, the user 401 does not need to wear the headphones 500, but only the direction sensor 510 and the user terminal 400a. The user terminal 400a wirelessly transmits the detection result by the direction sensor 510 to the voice processing device 200a.

  In FIG. 25, four speakers 330a to 330d are provided as an example. In order to localize the sound image in the left-right direction and the front-rear direction of the user 401, at least three speakers are provided.

  FIG. 26 is a block diagram illustrating an example of processing functions of the user terminal and the voice processing device according to the third embodiment. In FIG. 26, components corresponding to those in FIG. 6 are denoted by the same reference numerals, and description thereof is omitted.

The user terminal 400a includes the line-of-sight direction detection unit 421, but does not include the reproduction processing unit 422 of FIG.
The audio processing device 200a is different from the audio processing device 200 of FIG. 6 in the following points.

  A plurality of audio signals 251 stored in advance in the storage device 250 are input to the audio input unit 212. The plurality of audio signals 251 are sound sources associated with the exhibits, and the audio input unit 212 reads out the audio signal 251 from the storage device 250 and supplies it to the audio output processing unit 216. The storage device 250 may be a device installed outside the audio processing device 200a, or may be a device inside the storage device 250 (for example, the HDD 203).

  The audio output processing unit 216 outputs the synthesized audio signal to the speakers 330a to 330d instead of the user terminal. The audio output processing unit 216 determines the volume balance for each output channel of the audio signal 251 corresponding to each sound source from the position of each sound source in the virtual space and the user position Qt, and localizes the sound image of each sound source in space. Let Then, the audio output processing unit 216 controls the volume of the audio signal 251 corresponding to the sound source included in the listening range in the same procedure as in the second embodiment.

According to the third embodiment described above, the user 401 can easily recognize the sound corresponding to the desired exhibit by a natural operation, as in the second embodiment.
[Fourth Embodiment]
FIG. 27 is a diagram illustrating a configuration example of a voice providing system according to the fourth embodiment. In FIG. 27, components corresponding to those in FIG. 25 are denoted by the same reference numerals, and description thereof is omitted. Hereinafter, differences between the fourth embodiment and the third embodiment will be described.

  In the audio providing system according to the fourth embodiment, the user 401 listens to audio output from a plurality of speakers installed in the exhibition hall. However, in the fourth embodiment, unlike the third embodiment, each speaker is associated with an exhibit and installed at a position close to the corresponding exhibit. In the example of FIG. 27, the speakers 340a to 340c are set at positions close to the exhibits 310a to 310c, respectively.

  The voice processing device 200b basically has the same processing functions as the voice processing device 200a of the third embodiment, but differs from the voice processing device 200a of the third embodiment in the following points. The audio output processing unit 216 of the audio processing device 200b outputs an audio signal to a speaker associated with each exhibit. Here, when one sound source is associated with each exhibit, the sound output processing unit 216 of the sound processing device 200b outputs a sound signal corresponding to one sound source to an output channel to one speaker. There is no need to localize the sound image by adjusting the volume for each output channel. Therefore, the audio output processing unit 216 may simply control the volume of the audio signal 251 corresponding to the sound source included in the listening range in the same procedure as in the second embodiment.

According to the fourth embodiment described above, as in the second and third embodiments, the user 401 can easily recognize the sound corresponding to the desired exhibit by natural operation.
Note that the processing functions of the speech processing apparatus described in each of the above embodiments can be realized by a computer. In that case, a program describing the processing contents of the functions that each device should have is provided, and the processing functions are realized on the computer by executing the program on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic storage device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Examples of the magnetic storage device include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape. Examples of the optical disc include a DVD, a DVD-RAM, a CD-ROM, and a CD-R / RW. Magneto-optical recording media include MO (Magneto-Optical disk).

  When distributing the program, for example, a portable recording medium such as a DVD or a CD-ROM in which the program is recorded is sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. In addition, each time a program is transferred from a server computer connected via a network, the computer can sequentially execute processing according to the received program.

Regarding the above embodiments, the following supplementary notes are further disclosed.
(Supplementary Note 1) In a sound processing device that controls output of sound signals respectively corresponding to a plurality of virtual sound sources virtually arranged around a listener,
A state determination unit for determining whether the movement of the listener direction indicating the direction of the listener is in a stationary state;
The volume of the audio signal corresponding to the virtual sound source included in the listening range set so that the listener direction is the center when viewed from the listener, and the sound corresponding to the virtual sound source not included in the listening range An output control unit that controls to be relatively larger than the volume of the signal, and when it is determined that the stationary state is reached, an output control unit that reduces the listening range;
A speech processing apparatus comprising:

  (Supplementary note 2) The speech processing apparatus according to supplementary note 1, wherein when it is determined that the output control unit is in the stationary state, the listening range is reduced continuously or stepwise over time.

  (Additional remark 3) When it determines with the said output control part having become the said stationary state, the said listening range is reduced so that the said listening range may include at least 1 said virtual sound source, It is characterized by the above-mentioned. The speech processing apparatus according to Supplementary Note 1 or 2.

(Additional remark 4) It further has a distance detection part which detects the distance of the position of each virtual sound source or the position of the object matched with each virtual sound source, and the position of the listener,
When the output control unit reduces the listening range, the distance between the position of the virtual sound source included in the listening range or the position of the object associated with the virtual sound source and the position of the listener is shorter. Correcting the size of the listening range so that the listening range becomes larger;
The speech processing apparatus according to supplementary note 1 or 2, characterized by the above.

  (Additional remark 5) The said output control part, as for the said virtual sound source arrange | positioned in the position nearer to the center of the said listening range seeing from the said listener among the said virtual sound sources contained in the said listening range, The sound processing apparatus according to any one of appendices 1 to 4, wherein the volume is increased.

  (Additional remark 6) The said state determination part determines with having become the said stationary state, when the fluctuation amount of the said listener direction is settled in the predetermined fluctuation range for predetermined time, The additional notes 1-5 characterized by the above-mentioned The speech processing apparatus according to any one of the above.

  (Additional remark 7) The said output control part synthesize | combines the audio | voice signal corresponding to each of these virtual sound sources, produces | generates the synthetic | combination audio | voice signal of a predetermined number of channels, and transmits the said synthetic | combination audio | voice signal to a predetermined | prescribed audio | voice output apparatus. The speech processing apparatus according to any one of appendices 1 to 6, characterized in that:

(Additional remark 8) It is the audio | voice processing method in the audio | voice processing apparatus which controls the output of the audio | voice signal respectively corresponding to the some virtual sound source virtually arrange | positioned around a listener,
The volume of the audio signal corresponding to the virtual sound source included in the listening range set so that the listener's direction indicating the direction of the listener is the center when viewed from the listener is not included in the listening range. Control to be relatively larger than the volume of the audio signal corresponding to the virtual sound source,
When it is determined that the movement in the listener direction has become stationary, the listening range is reduced.
And a voice processing method.

(Supplementary note 9) The speech processing method according to supplementary note 8, wherein when it is determined that the stationary state is reached, the listening range is reduced continuously or stepwise over time.
(Supplementary note 10) The supplementary note 8 or 9, wherein when it is determined that the stationary state is reached, the listening range is reduced so that at least one virtual sound source is included in the listening range. Voice processing method.

(Additional remark 11) The process which further detects the distance of the position of each said virtual sound source or the position of the object matched with each said virtual sound source, and the position of the said listener,
In the process of reducing the listening range, the listening range becomes larger as the distance between the position of the virtual sound source included in the listening range or the position of the object associated with the virtual sound source and the position of the listener is shorter. So as to correct the size of the listening range,
The speech processing method according to appendix 8 or 9, characterized in that.

  (Supplementary note 12) Of the virtual sound sources included in the listening range, the volume of the corresponding audio signal is increased as the virtual sound source is located closer to the center of the listening range as viewed from the listener. The speech processing method according to any one of appendices 8 to 11, which is characterized by the following.

  (Additional remark 13) It determines with having become the said stationary state, when the variation | change_quantity of the said listener direction is settled in the predetermined fluctuation range only for predetermined time, It is determined in any one of Additional remark 8-12 characterized by the above-mentioned. Voice processing method.

  (Additional remark 14) It further includes the process which synthesize | combines the audio | voice signal corresponding to each of these virtual sound sources, produces | generates the synthetic | combination audio | voice signal of a predetermined number of channels, and transmits the said synthetic | combination audio | voice signal to a predetermined | prescribed audio | voice output apparatus. 14. The voice processing method according to any one of appendices 8 to 13, which is a feature.

(Supplementary Note 15) In an audio processing program for controlling output of audio signals respectively corresponding to a plurality of virtual sound sources virtually arranged around a listener,
On the computer,
The volume of the audio signal corresponding to the virtual sound source included in the listening range set so that the listener's direction indicating the direction of the listener is the center when viewed from the listener is not included in the listening range. Control to be relatively larger than the volume of the audio signal corresponding to the virtual sound source,
When it is determined that the movement in the listener direction has become stationary, the listening range is reduced.
A voice processing program for executing a process.

(Supplementary note 16) The audio processing program according to supplementary note 15, wherein when it is determined that the stationary state is reached, the listening range is reduced continuously or stepwise over time.
(Supplementary note 17) The supplementary note 15 or 16, wherein when it is determined that the listening state has been reached, the listening range is reduced so that the listening range includes at least one virtual sound source. Voice processing program.

(Additional remark 18) Let the said computer further perform the process which detects the distance of the position of each said virtual sound source or the position of the object matched with each said virtual sound source, and the position of the said listener,
In the process of reducing the listening range, the listening range becomes larger as the distance between the position of the virtual sound source included in the listening range or the position of the object associated with the virtual sound source and the position of the listener is shorter. So as to correct the size of the listening range,
The voice processing program according to supplementary note 15 or 16, characterized in that.

  (Supplementary Note 19) Of the virtual sound sources included in the listening range, the volume of the corresponding audio signal is increased as the virtual sound source is located closer to the center of the listening range as viewed from the listener. 19. The voice processing program according to any one of supplementary notes 15 to 18, which is a feature.

  (Supplementary note 20) According to any one of supplementary notes 15 to 19, wherein the stationary state is determined when the fluctuation amount in the listener direction is within a predetermined fluctuation range for a predetermined time. Voice processing program.

DESCRIPTION OF SYMBOLS 1 Audio processing apparatus 2 Sound source position information 3 State determination part 4 Output control part 10 Listener 21-25 Virtual sound source 30 Listening range

Claims (9)

In an audio processing device that controls output of audio signals respectively corresponding to a plurality of virtual sound sources virtually arranged around a listener,
A state determination unit for determining whether the movement of the listener direction indicating the direction of the listener is in a stationary state;
The volume of the audio signal corresponding to the virtual sound source included in the listening range set so that the listener direction is the center when viewed from the listener, and the sound corresponding to the virtual sound source not included in the listening range An output control unit that controls to be relatively larger than the volume of the signal, and when it is determined that the stationary state is reached, an output control unit that reduces the listening range;
A speech processing apparatus comprising:   The audio processing apparatus according to claim 1, wherein the output control unit reduces the listening range continuously or stepwise over time when it is determined that the stationary state is reached.   The output control unit reduces the listening range so that at least one of the virtual sound sources is included in the listening range when it is determined that the stationary state is reached. The speech processing apparatus according to 1 or 2. A distance detection unit for detecting a distance between the position of each virtual sound source or the position of an object associated with each virtual sound source and the position of the listener;
When the output control unit reduces the listening range, the distance between the position of the virtual sound source included in the listening range or the position of the object associated with the virtual sound source and the position of the listener is shorter. Correcting the size of the listening range so that the listening range becomes larger;
The speech processing apparatus according to claim 1 or 2,   The output control unit increases the volume of the corresponding audio signal for the virtual sound source arranged at a position closer to the center of the listening range when viewed from the listener among the virtual sound sources included in the listening range. The speech processing apparatus according to claim 1, wherein   The said state determination part determines with having become the said stationary state, when the fluctuation amount of the said listener direction is settled in the predetermined fluctuation range only for the predetermined time, The any one of Claims 1-5 characterized by the above-mentioned. The speech processing apparatus according to the item.   The output control unit synthesizes audio signals corresponding to each of the plurality of virtual sound sources to generate a synthesized audio signal having a predetermined number of channels, and transmits the synthesized audio signal to a predetermined audio output device. The speech processing apparatus according to any one of claims 1 to 6. An audio processing method in an audio processing device that controls output of audio signals respectively corresponding to a plurality of virtual sound sources virtually arranged around a listener,
The volume of the audio signal corresponding to the virtual sound source included in the listening range set so that the listener's direction indicating the direction of the listener is the center when viewed from the listener is not included in the listening range. Control to be relatively larger than the volume of the audio signal corresponding to the virtual sound source,
When it is determined that the movement in the listener direction has become stationary, the listening range is reduced.
And a voice processing method. In an audio processing program for controlling output of audio signals respectively corresponding to a plurality of virtual sound sources virtually arranged around a listener,
On the computer,
The volume of the audio signal corresponding to the virtual sound source included in the listening range set so that the listener's direction indicating the direction of the listener is the center when viewed from the listener is not included in the listening range. Control to be relatively larger than the volume of the audio signal corresponding to the virtual sound source,
When it is determined that the movement in the listener direction has become stationary, the listening range is reduced.
A voice processing program for executing a process.

Images