JP7463796B2 - DEVICE SYSTEM, SOUND QUALITY CONTROL METHOD AND SOUND QUALITY CONTROL PROGRAM - Google Patents

Description

One embodiment of the present invention relates to an acoustic device and a sound quality control method that changes the sound quality of a virtual sound source in response to the movement of an obstruction and the movement of the virtual sound source that is hidden by the obstruction.

An AR system has been proposed that allows a user to experience augmented reality (AR) by wearing an acoustic device such as headphones or earphones. The AR system emits sound from the acoustic device according to the location where the user is staying. The AR system detects the user's current position and the direction of the user's head in order to localize a virtual sound source at a predetermined localization position. The AR system localizes the virtual sound source at a predetermined position by applying specific signal processing to the sound using a head-related transfer function according to the detected position and head direction.

The head-related transfer function is the transfer function of sound from the sound source position to the ear canals of both user's ears. When sound generated at the sound source position reaches the user's ears, the frequency characteristics change with characteristics according to the direction of the sound source due to the head shape, auricle shape, etc. The head-related transfer function is a function that represents the frequency characteristics that change before the sound reaches the user's ears, and is prepared for each sound source direction. The user determines the direction from which the sound is coming by listening to the frequency characteristics specific to each sound source direction. Therefore, by processing the sound using the head-related transfer function for a specified direction and playing it back, the AR system can give the user the sensation that the sound is coming from the specified direction.

When there is a real or virtual obstruction between the user's position and the virtual sound source, it is preferable that the effect of the obstruction is reflected in the sound quality. For example, Patent Document 1 discloses a voice navigation system that navigates the user by presenting a route according to the location where the user is staying. This document proposes that when a guide voice for an object (destination) is played back, if there is an obstruction between the object and the user's position, the sound quality is changed (attenuated). Note that the system of Patent Document 1 attenuates the navigation voice when there is an obstruction between the object and the user's position, but does not localize the navigation voice to the object's position.

International Publication No. 2017/018298

Conventional AR systems do not control the sound quality of virtual sound sources by taking into account whether the location of the virtual sound source is blocked by a real or virtual door, window, or other obstruction. This causes a problem of reducing the realism of augmented reality.

Therefore, one of the objectives of one embodiment of the present invention is to make it possible to process the way a virtual sound source sounds more realistically by taking into account the state of obstruction of the virtual sound source by obstacles such as doors and windows in the real world.

A device system according to one embodiment of the present invention includes a device, a sensor, and a sound generating means. The device is worn by a user. The sensor detects the movement of a movable obstruction. The sound processing unit generates sound of a first sound quality when a virtual sound source located on the opposite side of the obstruction is obstructed by the obstruction and emits the sound from the device, and when the sensor detects that the obstruction has moved from a position obstructing the virtual sound source, changes the sound quality of the sound from the sound quality when obstructed by the obstruction to a second sound quality when not obstructed.

A sound quality control method according to one embodiment of the present invention is characterized in that a device system including an acoustic device worn by a user and a sensor for detecting the movement of a movable obstruction generates sound of a first sound quality when obstructed by an obstruction from a virtual sound source positioned on the opposite side of the obstruction, emits the sound from the acoustic device, and when the sensor detects that the obstruction has moved, changes the sound quality of the sound from the first sound quality to a second sound quality when not obstructed by an obstruction.

A sound quality control program according to one embodiment of the present invention is characterized in that it causes a control unit of a mobile terminal device that communicates with an acoustic device worn by a user and a sensor that detects the movement of a movable obstruction to function as a sound control means that generates sound of a first sound quality when obstructed by an obstruction from a virtual sound source positioned on the opposite side of the obstruction, emits the sound from the acoustic device, and when the sensor detects that the obstruction has moved, changes the sound quality of the sound from the first sound quality to a second sound quality when not obstructed by an obstruction.

According to one embodiment of the present invention, it is possible to process the way a virtual sound source sounds more realistically by taking into account the state of obstruction of the virtual sound source by an obstruction object.

FIG. 1 is a diagram showing the configuration of an audio reproduction system according to an embodiment of the present invention. FIG. 2 is a block diagram of a portable terminal device of the audio reproduction system. FIG. 3 is a block diagram of a headphone of the audio reproduction system. FIG. 4 is a block diagram of a door sensor of the audio playback system. FIG. 5 is a plan view of a building in which the audio reproduction system is used. FIG. 6 is a diagram for explaining indirect sound in a sound reproduction system. FIG. 7 is a diagram for explaining a direct sound in an audio reproduction system. FIG. 8 is a diagram for explaining transmitted sound in an audio reproduction system. FIG. 9 is a diagram showing the configuration of a signal processing unit of the portable terminal device. FIG. 10 is a flowchart showing the processing operation of the mobile terminal device. FIG. 11 is a flowchart showing the processing operation of the mobile terminal device. FIG. 12 is a flowchart showing the processing operation of the mobile terminal device. FIG. 13 is a plan view of a building in an audio reproduction system in which a sound source moves.

Fig. 1 is a diagram showing the configuration of an audio reproduction system 1 to which the present invention is applied. Fig. 2 is a block diagram of a mobile terminal device 10 of the audio reproduction system 1. The audio reproduction system 1 includes the mobile terminal device 10, headphones 20 which are an acoustic device, and a door sensor 30. Fig. 1 shows an example in which a user L holds the mobile terminal device 10 in his/her hand and wears the headphones 20. With this equipment, the user L enters a room 201 shown in Fig. 5. When the user L enters the room 201, the mobile terminal device 10 reproduces audio localized in the room 202 based on a scenario file 72 (also simply referred to as the scenario 72). The headphones 20 correspond to the acoustic device of the present invention.

The mobile terminal device 10 is, for example, a smartphone (multi-function mobile phone). The mobile terminal device 10 and the headphones 20 are connected by Bluetooth (registered trademark) and can communicate with each other. The connection between the mobile terminal device 10 and the headphones 20 is not limited to Bluetooth, and may be other wireless communication standards or wired. The headphones 20 are a so-called ear-hook type that combines two speakers 21R and 21L with a headband 22. The headphones 20 have a posture sensor 23 on the headband 22 and can track the orientation of the user L's head. The posture sensor 23 may be any of a three-axis gyro (angular velocity) sensor, a six-axis sensor (three-axis gyro sensor + three-axis motion (acceleration) sensor), and a nine-axis sensor (three-axis gyro sensor + three-axis motion sensor + three-axis compass (direction) sensor). An earphone may be used as the acoustic device instead of the headphones 20.

The door sensor 30 detects the open/closed state of a door 502 provided on a fixed wall 501 (see FIG. 5). The door sensor 30 transmits information on the open/closed state of the door 502 to the mobile terminal device 10. The door sensor 30 and the mobile terminal device 10 are connected by Bluetooth Low Energy (BLE). BLE and normal Bluetooth can be used together in one device. In this embodiment, Bluetooth is used to connect the mobile terminal device 10 and the headphones 20, as described above. The connection between the door sensor 30 and the mobile terminal device 10 is not limited to BLE. For example, the door sensor 30 and the mobile terminal device 10 may be connected via the Internet, such as Wi-Fi (registered trademark) or a mobile communication network.

Figure 2 is a block diagram of the mobile terminal device 10. In terms of hardware, the mobile terminal device 10 is a smartphone equipped with a control unit 100, a memory unit 103, a voice generation unit 105, a signal processing unit 106, and a communication processing unit 107. The control unit 100 includes a CPU. The memory unit 103 includes a ROM, a RAM, and a flash memory.

The mobile terminal device 10, the headphones 20, and the door sensor 30 function as the audio playback system 1 when the mobile terminal device 10 activates the program 70 stored in the memory unit 103.

The storage unit 103 stores the above-mentioned program 70, as well as a scenario file 72 and audio data 73. The program 70 is an application program for causing the mobile terminal device 10, the headphones 20, and the door sensor 30 to function as an audio playback system. The scenario file 72 is a file in which a procedure for sequentially playing back predetermined audio for the user L is described. The scenario file 72 includes a layout table 72A in which the location where the audio is to be played back, for example, the shape of the building 200 and the arrangement of the walls 500 shown in FIG. 5, are described. The audio data 73 is data of audio played back according to the scenario file 72. The audio data 73 may be, for example, an audio signal such as PCM or MP4, or may be audio synthesis data using the audio generation unit 105 as a synthesizer. When the audio data 73 is an audio signal, the control unit 100 may read out the audio data 73 as the audio generation unit 105.

The filter coefficients 71 include a head-related transfer function and an impulse response at a predetermined position in the room 201. These filter coefficients are used to localize the virtual sound source SP1 shown in FIG. 5 etc. at a predetermined position relative to the user L. The filter coefficients 71 are used in the signal processing unit 106. In the following description, SP1 is also referred to as the virtual sound source position.

The control unit 100 controls the operation of the mobile terminal device 10. When the program 70 is started, the control unit 100 also functions as a position determination unit 101 and a head direction determination unit 102. The position determination unit 101 determines the current position of the mobile terminal device 10. In the audio reproduction system 1, the position of the mobile terminal device 10 is used as the position of the user L. If the position determination unit 101 is outdoors, a satellite positioning system such as GPS or Michibiki may be used. If the position determination unit 101 is indoors, a position measurement system such as a beacon installed indoors may be used. Even if indoors, the position determination unit 101 may first determine an accurate position using a satellite positioning system outdoors (calibration), and then use the attitude sensor 23 to trace the movement of the user L to determine the indoor position. When tracing the movement of the user L, the attitude sensor 23 is preferably a 6-axis sensor or 9-axis sensor that can detect the motion of the user L.

The head direction determination unit 102 determines the orientation of the user L's head based on the detection value of the orientation sensor 23 acquired from the headphones 20. If the orientation sensor 23 is a three-axis gyro sensor, the control unit 100 first determines the head direction by having the user L face a specified direction (calibration). The control unit 100 then acquires angular velocity information from the orientation sensor 23 and determines the current orientation of the user L's head by integrating this angular velocity information with the head direction. If the orientation sensor 23 includes a gyro sensor and a compass sensor, it is sufficient to use the fast-response gyro sensor to follow changes in the head direction of the user L, while canceling the integral error of the gyro sensor with the slow-response compass sensor.

The sound generation unit 105 plays back the sound data 73. The sound data 73 may be a sound signal such as PCM or MP4, or may be sound synthesis data using the sound generation unit 105 as a synthesizer. The signal processing unit 106 controls the quality of the sound according to the position of the virtual sound source SP1 (see FIG. 5) and the user L. The signal processing unit 106 also determines the localization of the sound according to the position and orientation of the user L acquired from the position determination unit 101 and the head direction determination unit 102, reads out the head related transfer function of the localization direction, and filters the sound. The sound generation unit 105 and the signal processing unit 106 correspond to the sound processing unit of the present invention.

Based on the scenario file 72, the audio reproduction system 1 localizes audio to the adjacent room 202 separated by a wall 500 for the user L in the room 201 (see FIG. 5-8, etc.). The wall 500 is provided with a door 502. When the door 502 is closed, the audio reproduction system reproduces audio with a sound quality that sounds like the audio is coming from the other room blocked by the wall 500. When the door 502 is open, the audio reproduction system reproduces audio with a sound quality that sounds like the audio reverberating in the room 202 is coming through the door frame 503. The mobile terminal device 10 (communication processing unit 107) transmits the processed audio to the headphones 20. The headphones 20 output the received audio from the speakers 21R and 21L. This allows the user L to hear the audio with an auditory sensation of coming from a predetermined localization position according to the scenario file 72.

The communication processing unit 107 communicates with the headphones 20 and the door sensor 30, which are Bluetooth-compatible devices. The communication processing unit 107 communicates with the communication processing unit 24 of the headphones 20 via Bluetooth. The communication processing unit 107 communicates with the communication processing unit 32 of the door sensor 30 via Bluetooth BLE. The communication processing unit 107 transmits an audio signal to the headphones 20 and receives the detection value of the attitude sensor 23 from the headphones 20. The communication processing unit 107 receives information on the open/closed state of the door 502 from the door sensor 30.

Figure 3 is a block diagram showing the configuration of the headphones 20. The headphones 20 include speakers 21L and 21R, a position sensor 23, a communication processing unit 24, an AIF 25, DACs 26L and 26R, and amplifiers 27L and 27R.

The communication processing unit 24 communicates with the mobile terminal device 10 (communication processing unit 107) via Bluetooth or BLE. The AIF (Audio Interface) 25 separates the audio signal received from the mobile terminal device 10 into left and right channel signals and transmits them to the DACs 26L and 26R. The DACs (Digital to Analog Converters) 26L and 26R convert the digital signal input from the AIF 25 into an analog signal. The amplifiers 27L and 27R amplify the analog signal input from the DACs 26L and 26R and supply it to the speakers 21L and 21L. As a result, the audio signal received from the mobile terminal device 10 is emitted as sound from the speakers 21L and 21R. Since the headphones 20 are worn on the head of the user L, the sound emitted from the speakers 21 and 21R is heard by the left and right ears of the user L.

Figure 4 is a block diagram of the door sensor 30. As shown in Figure 5, the door sensor 30 is attached near the hinge of the door 502, and detects and outputs information on the open/closed state of the door 502 relative to the fixed wall 501. The information on the open/closed state is expressed as an angle relative to the fixed wall 501. The door sensor 30 includes a sensor module 31 and a communication processing unit 32. The sensor module 31 detects the open/closed state of the door 502. The sensor module 31 is composed of, for example, a rotary encoder, a semiconductor sensor, or a photoelectric sensor. The rotary encoder rotates coaxially with the hinge of the door 502 to detect the rotation angle or absolute angle of the door 502. The semiconductor sensor detects the angular velocity due to the opening and closing of the door 502, and calculates the angle of the door 502 by integrating this angular velocity. The optical sensor has a light emitting unit and a light receiving unit on the door 502 and the fixed wall 501, respectively, and detects the angle of the door 502 by the change in the position of the light from the light emitting unit to the light receiving unit. The sensor module 31 is not limited to the above. For example, the sensor module 31 may be a potentiometer. Also, if the sensor module 31 only needs to detect whether the door 502 is completely closed or slightly open, the sensor module 31 may be a limit switch. The communication processing unit 32 transmits information on the open/closed state of the door 502 detected by the sensor module 31 to the mobile terminal device 10.

Figure 5 is a plan view of a building 200 into which a user L is guided and a scenario is executed by the audio playback system of the present invention. Building 200 is provided with rooms 201 and 202. Rooms 201 and 202 are separated by a wall 500. Wall 500 has a fixed wall 501 and a door 502 provided in a part of fixed wall 501. Door 502 is attached to a door frame 503 formed on fixed wall 501. Door 502 can freely swing towards room 201. Wall 500 corresponds to a shield of the present invention.

The building 200 and positions within it are specified by XY coordinates. As shown in the lower left of FIG. 5, the XY coordinates are determined based on the X axis set in the left-right direction in the figure and the Y axis set in the up-down direction. The shapes of the rooms 201 and 202, the positions of the walls 500, and the doors 502 are all expressed in XY coordinates and stored in the layout table 72A. The audio reproduction system 1 of this embodiment performs audio localization processing in two dimensions, that is, performs sound image localization processing by assuming that the positions of the sound source and the ears of the user L are all at the same height. If sound image localization is performed in three dimensions including height, the Z axis in the height direction can be set in the front-to-back direction of the drawing.

The door 502 is provided with a door sensor 30 for detecting whether the door 502 is open or closed. In FIG. 5, the door 502 is open toward the room 201. The door 502 may be opened manually by the user L, or may be opened automatically by an actuator (not shown).

User L listens to the sound reproduced by the sound reproduction system 1 while moving around the room 201. The sound reproduction system 1 refers to the scenario file 72 based on the position, time, etc. of the user L, and reproduces the sound specified by the scenario file 72. In the scene shown in FIG. 5, the sound reproduction system 1 reproduces the sound of a piano performance localized at a virtual sound source position SP1 in the room 202. In FIG. 5, an actual piano 300 is placed at the location of the virtual sound source position SP1, but an actual piano 300 is not required.

In FIG. 5, when user L hears the sound of a piano being played through headphones 20, he moves to position LP1 near door 502 and opens door 502 (FIG. 5 shows door 502 open). This allows user L to recognize that piano 300 is playing in room 202, but user L cannot directly see piano 300 (virtual sound source position SP1) from position LP1 because fixed wall 501 blocks the view. After opening door 502, user L moves to position LP2 to find the location where the sound is being played. Having moved to position LP2, user L finds piano 300 (virtual sound source position SP1) by looking inside room 202 through door frame 503.

When the user L behaves as described above, the sound quality of the sound (piano performance sound) controlled by the sound reproduction system 1 is as follows. When the door 502 is closed, the sound reproduction system 1 reproduces the piano performance sound with a sound quality that makes it seem as if the piano 300 is being played on the other side of the wall 500 and is leaking through the wall 500 (door 502) for the user L. When the door 502 is open, the sound reproduction system 1 reproduces the piano performance sound with a sound quality that makes it seem as if the piano performance sound reverberating in the room 202 is being heard through the door frame 503. However, at position LP1, the user L cannot hear the direct sound of the piano performance sound localized at the virtual sound source position SP1. After that, the user L moves to position LP2 where the piano 300 is visible. When the user L moves to position LP2, the sound reproduction system 1 reproduces the direct sound of the piano performance sound for the user L.

With reference to Figures 6-8, the transmission mode of sound generated at virtual sound source position SP1 when door 502 is closed, when door 502 is open and user L is at position LP1, and when user L moves to position LP2 will be described. In the following description, the sound generated at virtual sound source position SP1 will be referred to as sound S (SP1). Note that in the descriptions in Figures 5-9 and 13, parentheses "( )" around sound S (SP1), etc. will be replaced with "-".

Figure 6 is a diagram explaining the sound (indirect sound) heard by the user L through the door frame 503 when the door 502 is open. The sound S (SP1) is transmitted throughout the room 202 and forms a reverberation in the room 202 by, for example, reflecting off the walls. The sound reverberating at position SP2 also sounds at position SP2 of the door frame 503. Hereinafter, the sound heard at position SP2 is referred to as sound S (SP2). The propagation of sound from the virtual sound source position SP1 to position SP2 is represented by an impulse response measured by placing a sound source at the virtual sound source position SP1 and a microphone at position SP2. Hereinafter, this impulse response is referred to as impulse response IR (1-2). As described above, impulse response IR (1-2) represents the response waveform when the piano performance sound S (SP1) reverberating in the room 202 is heard at position SP2. Audio S(SP2) is obtained by filtering audio S(SP1) with an FIR filter that convolves the impulse response IR(1-2).

When the door 502 is open, the sound S (SP2) that has reached the door frame 503 is transmitted to the room 201 and reaches the user L. The propagation of the sound from the position SP2 of the door frame 503 to both ears of the user L is represented by a head-related transfer function according to the position of the user L and the direction of the user L's head. The head-related transfer function at this time is hereinafter referred to as the head-related transfer function HRTF (2-L). The indirect sound S (L open), which is the sound that the user L hears from the open door 502 (door frame 503), is obtained by processing the sound S (SP2) with the head-related transfer function HRTF (2-L). Specifically, the indirect sound S (L open) is obtained by filtering the head impulse response, which is a time-domain coefficient sequence of the head-related transfer function HRTF (2-L), with an FIR filter that convolves the sound S (SP2) with the head impulse response. Note that, in order to simplify the processing, the reverberation (impulse response) in the room 201 is not taken into consideration.

Figure 7 is a diagram explaining the sound (direct sound) that the user L hears directly from the sound source. When the user L is at position LP2, the virtual sound source position SP1 is directly visible, and the direct sound of the sound S (SP1) is heard. The propagation of the direct sound is represented by a head-related transfer function according to the position of the user L from the virtual sound source position SP1 and the direction of the user L's head. Hereinafter, the head-related transfer function at this time is referred to as the head-related transfer function HRTF (1-L). The direct sound S (L direct) is obtained by processing the sound S (SP1) with the head-related transfer function HRTF (1-L). Specifically, the direct sound S (L direct) is obtained by filtering the sound S (SP1) with an FIR filter that convolves the head impulse response, which is a time-domain coefficient sequence of the head-related transfer function HRTF (1-L), with the sound S (SP1).

Figure 8 is a diagram explaining the sound (transmitted sound) that is heard by the user L through a closed door 502. In this embodiment, it is assumed that the fixed wall 501 does not transmit sound at all. When the door 502 is closed, the user L in the room 201 hears the sound that travels from the room 202 through the door 502. The sound that reaches the door 502 is the sound S (SP2) as described above. The transmitted sound S (L door) is the sound S (SP2) that reaches the door 502, passes through the closed door 502, travels to the door surface (room 201 side) SP20, and is transmitted from the door surface SP20 to the user L. Therefore, the propagation of the sound S (L door) is expressed by the following three impulse responses. The impulse response from the virtual sound source position SP1 to the door 502 (SP2), the impulse response from the door 502 (SP2) to the door surface SP20, and the head-related transfer function HRTF (20-L) from the door surface SP20 to the user L. Note that, to simplify the processing, the reverberation (impulse response) in the room 201 is not taken into consideration.

The head-related transfer function HRTF(20-L) is considered to be approximately the same as the head-related transfer function HRTF(2-L) in FIG. 6. Therefore, the head-related transfer function HRTF(2-L) may be used as the head-related transfer function HRTF(20-L).

The impulse response from door 502 (SP2) to door surface SP20 is the sound insulation characteristic of door 502. Hereinafter, the impulse response of the sound insulation characteristic of door 502 is referred to as impulse response IR (door).

Indirect sound S (L door), which is sound that passes through the closed door 502 and is heard by the user L, is obtained by processing the sound S (SP20) with a head-related transfer function HRTF (20-L). Specifically, the transmitted sound S (L door) is obtained by filtering with an FIR filter that convolves the head impulse response, which is a time-domain coefficient sequence obtained by converting the head-related transfer function HRTF (20-L), with the sound S (SP20).

As shown in FIG. 8, when the door 502 is closed, the audio reproduction system 1 reproduces only the transmitted sound S (L door) for the user L.

As shown in FIG. 6, when the door 502 is open but the user L is in a location (e.g., location LP1) where the piano 300 is not visible, the audio playback system 1 plays for the user L an indirect sound S (L open) heard from the door frame 503.

As shown in FIG. 7, when the door 502 is open and the user L is in a position where the piano 300 can be seen (e.g., position LP2), the sound reproduction system 1 reproduces for the user L a direct sound S (L direct) from the virtual sound source position SP1 (piano 300) and an indirect sound S (L open). This is because the indirect sound S (L open) can be heard by the user L even if the user L is in a position where the piano 300 can be seen.

The gain of the direct sound S (L direct) may be changed depending on whether the user L is in a position where the piano 300 (virtual sound source position SP1) is completely visible or only partially visible. In this case, the sound quality may be adjusted in the frequency domain, such as by slightly attenuating the high-pitched sounds.

Figure 9 is a functional block diagram of the signal processing unit 106. The signal processing unit 106 is composed of, for example, a DSP (digital signal processor), and various functional units are configured by a program to perform signal processing of the sound generated by the sound generation unit 105. As described above, the sound generation unit 105 generates sounds such as piano performance sounds. The signal processing unit 106 processes the sounds to generate transmitted sound S (L door), indirect sound S (L open), and direct sound S (L direct). The filters 64-69 shown in the figure are all FIR filters. In Figure 9, the signal flow is represented by a single line, but the sound signals to be processed are two-channel signals, left and right.

The serially connected filters 64-66 generate a transmitted sound S (L door) when the door 502 is closed. The filter 64 is set with an impulse response IR (1-2) from the virtual sound source position SP1 to the position SP2 of the door 502. The filter 64 filters the sound S (SP1) to generate a sound S (SP2). The filter 65 is set with an impulse response IR (door) which is the sound insulation characteristic of the door 502. The filter 65 filters the sound S (SP2) to generate a sound S (SP20). The filter 66 is set with a head-related transfer function (head impulse response) HRTF (20-L) according to the position and head direction of the user L from the position SP20 on the room 201 side of the door 502. The filter 66 filters the sound S (SP20) to generate a transmitted sound S (L door). In this embodiment, three filters 64-66 are provided to calculate the impulse response IR(1-2), the impulse response (sound insulation characteristics) IR(door), and the head-related transfer function HRTF(20-L). However, the transmitted sound S(L door) may be generated by a single filter that combines these filter coefficients.

The serially connected filters 67 and 68 generate an indirect sound S (L open) when the door 502 is open. An impulse response IR (1-2) from the virtual sound source position SP1 to the position SP2 of the door 502 is set in the filter 67. The filter 67 filters the sound S (SP1) to generate the sound S (SP2). A head-related transfer function (head impulse response) HRTF (2-L) according to the position of the user L and the direction of the head from the position SP2 of the door 502 is set in the filter 68. The filter 68 filters the sound S (SP2) to generate the indirect sound S (L open). In this embodiment, two filters 66 and 67 are provided for the impulse response IR (1-2) and the head-related transfer function HRTF (2-L). However, the indirect sound S (L open) may be generated by a single filter that combines these filter coefficients.

The filter 69 generates a direct sound S (L direct). A head-related transfer function (head impulse response) HRTF (1-L) corresponding to the position of the user L from the virtual sound source position SP1 and the direction of the head of the user L is set in the filter 68. The filter 69 filters the sound S (SP1) to generate the direct sound S (L direct).

The gain adjustment units 61-63 turn on/off and adjust the gain of the generated transmitted sound S (L door), indirect sound S (L open), and direct sound S (L direct). Since the impulse response and head-related transfer function include an element of volume control, the signal processing unit 106 does not usually need to adjust the gain after generating the transmitted sound S (L door), etc. The gain adjustment units 61-63 are used when adjusting the gain of the indirect sound S (L open) according to the opening angle of the door 502, when cross-fading the transmitted sound S (L door) and the indirect sound S (L open), etc.

The adder 80 adds the transmitted sound S (L door), the indirect sound S (L open), and the direct sound S (L direct) whose gains have been adjusted by the gain adjustment units 61-63, to generate the sound S (L) to be output to the headphones 20. The signal processing unit 106 inputs the sound S (L) to the communication processing unit 107. The communication processing unit 107 transmits the sound S (L) to the headphones 20. The signal processing unit 106 can also calculate one filter coefficient by combining the filter coefficients and gain values of all the filters 64-69 and all the gain adjustment units 61-63. The signal processing unit 106 can also generate the sound S (L) with one FIR filter using this filter coefficient.

Figure 10 is a flowchart showing the audio signal processing operation of the mobile terminal device 10. This process is executed when audio is being generated based on the scenario file 72. This process is executed periodically, for example every 20 milliseconds, by the control unit 100 and signal processing unit 106 of the mobile terminal device 10.

The control unit 100 acquires the current position and head direction of the user L (steps S11, S12). Hereinafter, in this flowchart, step Sn (n is an arbitrary numerical value) will be simply referred to as Sn. The control unit 100 judges the signal received from the door sensor 30 and judges whether the door 502 is open or not (S13). If the door 502 is closed (NO in S13), the user L will only hear the transmitted sound, and therefore instructs the signal processing unit 106 shown in FIG. 9 to generate the transmitted sound S (L door) (S14). The transmitted sound S (L door) output from the signal processing unit 106 is output to the communication processing unit 107 (S15).

If the door 502 is open (YES in S13), the control unit 100 determines whether the user L is in a position where the virtual sound source position SP1 (piano 300) can be viewed directly (S21). If the user L is in a position where the virtual sound source position SP1 can be viewed directly (YES in S21), the control unit 100 advances the process to S25. If the user L is in a position where the virtual sound source position SP1 cannot be viewed directly (NO in S21), the control unit 100 advances the process to S22.

If the user L is in a position where he/she cannot look directly at the virtual sound source position SP1 (NO in S21), the control unit 100 instructs the signal processing unit 106 to generate an indirect sound S(L open) (S22). At this time, the control unit 100 selects one head-related transfer function based on the position and head direction of the user L, and sets it in the filter 68. The indirect sound S(L open) output from the signal processing unit 106 is output to the communication processing unit 107 (S15).

When the door 502 is open and the user L is in a position where he or she can look directly at the virtual sound source position SP1 (YES in S21), the control unit 100 instructs the signal processing unit 106 to generate a direct sound S (L direct) and an indirect sound S (L open) (S25, S26). At this time, the control unit 100 sets one head-related transfer function to each of the filters 68 and 69 based on the position and head direction of the user L. The signal processing unit 106 adds the generated direct sound S (L direct) and indirect sound S (L open) to generate a sound S (L) (S27) and outputs it to the communication processing unit 107 (S27).

In step S13 of FIG. 10, the control unit 100 determines whether the door 502 is open or closed. If the door 502 is open, the control unit 100 may determine the angle at which the door 502 is open and adjust the gain of the indirect sound S (L open) according to the angle. Furthermore, the control unit 100 may adjust the sound quality of the indirect sound S (L open) according to the opening angle of the door 502.

Figure 11 is a flowchart showing the process of adjusting the gain of the indirect sound S (L open) depending on the opening angle of the door 502. The control unit 100 acquires the opening angle of the door 502 from the door sensor 30 (S31). Based on the acquired opening angle, the control unit 100 sets a gain corresponding to this opening angle in the gain adjustment unit 62 (S32). At this time, the control unit 100 may perform a process (cross-fade) of increasing the gain of the indirect sound S (L open) and decreasing the gain of the transmitted sound S (L door) as the opening angle of the door 502 increases.

In step S21 of FIG. 10, the control unit 100 determines whether the user L is in a position where he or she can look directly at the virtual sound source SP1. The control unit 100 may give the virtual sound source SP1 a predetermined size, such as a piano 300, and adjust the gain of the direct sound S (L direct) depending on how directly the user L can look at the virtual sound source SP1. Furthermore, the control unit 100 may adjust the sound quality of the direct sound S (L direct) depending on how directly the user L can look at the virtual sound source SP1.

Figure 12 is a flowchart showing the process of adjusting the gain of the direct sound S (L direct) depending on the extent to which the user L can view the virtual sound source SP1 directly. The control unit 100 calculates the extent to which the virtual sound source SP1 can be viewed directly from the position of the user L (S33). The direct viewing range is calculated based on the coordinates of the user L, the virtual sound source SP1, and the door frame 503. The control unit 100 sets a gain in the gain adjustment unit 62 depending on the calculated direct viewing range (S34). That is, when the user L can view the entire virtual sound source SP1 directly, the control unit 100 sets a gain of 100%, and sets a smaller gain as the range of the virtual sound source SP1 that the user L can view directly becomes narrower.

The above embodiment describes a case where the virtual sound source SP1 (piano 300) does not move. The following embodiment describes a case where the virtual sound source SP1 moves. In this embodiment, parts with configurations similar to those in the above embodiment are given the same numbers and will not be described.

Figure 13 is a diagram showing the arrangement of a virtual sound source SP10 and a user L in a building 200. The layout of the room is the same as that shown in Figure 5. In Figure 13, the virtual sound source SP10, depicted with the appearance of a bird, moves from position SP10(1) to position SP10(2). The door 502 is open. In the embodiment of Figures 6 and 7, the user L moves so that he or she can look directly at the virtual sound source SP1. In this embodiment, the virtual sound source SP10 moves so that the user L can look directly at the virtual sound source SP10.

The user L remains at position LP10. When the virtual sound source SP10 is at position SP10(1), the user L cannot look directly at the virtual sound source SP10, and hears indirect sound S (L open). The indirect sound S (L open) is the same as that described in FIG. 6, and is calculated using the impulse response of the virtual sound source SP10(1) at SP2 and the head-related transfer function from SP2 to the user L. When the virtual sound source SP10 moves to position SP10(2), the user L can look directly at the virtual sound source SP10 through the door frame 503. The user L hears direct sound S (L direct). The direct sound S (L direct) is the same as that described in FIG. 7, and is calculated using the head-related transfer function from SP10(2) to the user L. In addition, when the direct sound is heard, the indirect sound is heard in parallel. This indirect sound is calculated using the impulse response of the virtual sound source SP10(2) in SP2 and the head-related transfer function from SP2 to the user L.

When the virtual sound source SP10 moves, the control unit 100 acquires the position of the virtual sound source SP10 in parallel with S10 and S11 in the flowchart of FIG. 10, and calculates whether the user L can look directly at the virtual sound source SP10.

The example in FIG. 13 shows an example in which the user L is stationary. Furthermore, an embodiment in which the user L moves in the same manner as in FIG. 5-8 and the virtual sound source SP10 also moves can also be realized.

The door 502 in this embodiment is of a swinging type that uses hinges. The door 502 is not limited to a swinging type, but also includes doors that open and close using other mechanisms, such as a sliding door. It is also possible to realize an embodiment in which the door 502 is always open, and an embodiment without the door 502, i.e., an embodiment with only a door frame 503 (opening).

The example in FIG. 5 was for a case where user L opens door 502. An embodiment in which user L closes an open door 502 is also possible.

The example in Figure 8 is an embodiment in which transmitted sound can be heard only through the door 502 when the door 502 is closed. It is also possible to realize an embodiment in which transmitted sound can be heard through the entire wall 500 when the door 502 is closed.

In the above embodiment, the audio generation unit 105 and the signal processing unit 106 are provided in the mobile terminal device 10. The audio generation unit 105 and the signal processing unit 106 may also be provided in the headphones 20.

In the above embodiment, an example has been described in which the building 200, the rooms 201 and 202, the walls 500, the doors 502, etc. actually exist. When the present invention is applied to VR (virtual reality), the building 200, the rooms 201 and 202, the walls 500, the doors 502, etc. may be virtual.

In the above embodiment, a head-related transfer function is used as an audio processing unit that changes the sound quality to the second sound quality when the sound is not blocked by an obstruction. However, for example, a process of changing the volume is also an example of the audio processing.

From the embodiment described above in detail, the following aspects can be understood:

<<Aspect 1>>
A device system according to a first aspect of the present disclosure includes a device, a sensor, and a sound generating means. The device is worn by a user. The sensor detects the movement of a movable obstruction. The sound processing unit generates a sound of a first sound quality when a virtual sound source located on the opposite side of the obstruction is obstructed by the obstruction and emits the sound from the device, and when the sensor detects that the obstruction has moved from a position obstructing the virtual sound source, changes the sound quality of the sound from the sound quality when obstructed by the obstruction to a second sound quality when not obstructed.

Aspect 2
In the device system according to the second aspect of the present disclosure, when at least one of the obstruction, the user, and the virtual sound source moves and the user is able to directly view the localization position of the virtual sound source, the device system generates sound of a third sound quality for the case where the localization position of the virtual sound source can be directly viewed, instead of the second sound quality. This allows the user to hear the direct sound.

Aspect 3
The device system according to the third aspect of the present disclosure uses, as the shielding object, a wall separating a first space in which a user is present from a second space in which a virtual sound source is located, and a door provided on the wall. As the sensor, a sensor that detects the opening and closing of the door is used. This realizes sound quality equivalent to that when listening to sound across a room.

Aspect 4
The device system according to the fourth aspect of the present disclosure uses a sensor that detects the degree of opening and closing of the door. The audio processing unit generates audio with a sound quality obtained by cross-fading the first sound quality and the second sound quality according to the degree of opening and closing of the door detected by the sensor. This realizes a change in sound quality when the door 502 is opened slightly or gradually.

Aspect 5
The device system according to aspect 5 of the present disclosure achieves a second sound quality by filtering the impulse response in a second space of a virtual sound source at position SP2 of the open door and a head-related transfer function from position SP2 to the user.

1 Audio playback system 10 Mobile terminal device (smartphone)
20 Headphones 21 Speaker 23 Attitude sensor 24 Communication processing unit 30 Door sensor 31 Sensor module 30 Communication processing unit 70 Application program 100 Control unit 106 Signal processing unit 500 Wall 501 Fixed wall 502 Door 503 Door frames SP1, SP10 Virtual sound source

Claims (11)

An acoustic device worn by a user;
A sensor for detecting the movement of an openable/ closable obstruction;
a sound processing unit that generates a sound of a first sound quality from a virtual sound source located on the opposite side of the obstruction when the virtual sound source is obstructed by the obstruction, emits the sound from the acoustic device, and changes the sound quality of the sound from the first sound quality to a second sound quality when the virtual sound source is not obstructed by the obstruction when the sensor detects that the obstruction has moved from a position obstructing the virtual sound source;
A device system comprising:
When at least one of the obstruction, the user, and the virtual sound source moves and the user is able to directly view the localization position of the virtual sound source, a sound of a third sound quality for when the user can directly view the localization position of the virtual sound source is generated instead of the second sound quality.
Device system. the obstruction is a wall that separates a first space in which the user is present from a second space in which the virtual sound source is located, and a door provided on the wall;
The device system according to claim 1 , wherein the sensor detects whether the door is open or closed. The sensor detects the degree of opening and closing of the door,
The device system according to claim 2 , wherein the audio processing unit generates audio having a sound quality obtained by cross-fading the first sound quality and the second sound quality in accordance with the degree of opening and closing of the door detected by the sensor. The second sound quality is
an impulse response in a second space of the virtual sound source at the position of the open door;
A head-related transfer function from the position of the open door to the user;
The device system according to claim 2 or 3 , wherein the sound quality is filtered by The first sound quality is a sound quality filtered by an impulse response of the virtual sound source in the second space at the position of the open door, an impulse response of the sound insulation characteristic of the obstruction, and a head-related transfer function from the position of the open door to the user;
The third sound quality is a sound quality filtered by a head-related transfer function from the localization position of the virtual sound source to the user.
The device system according to claim 4 . A device system including an acoustic device worn by a user and a sensor for detecting movement of a movable obstruction,
A sound quality control method comprising: generating a sound of a first sound quality when a virtual sound source is blocked by the blocking object from a virtual sound source located on an opposite side of the blocking object, emitting the sound from the acoustic device; and changing a sound quality of the sound from the first sound quality to a second sound quality when the virtual sound source is not blocked by the blocking object when the sensor detects that the blocking object has moved from a position blocking the virtual sound source.
When the user is in a position where the user can directly view the localization position of the virtual sound source, a sound of a third sound quality for when the user can directly view the localization position of the virtual sound source is generated instead of the second sound quality.
Sound quality control method . As the shield, a wall separating a first space in which the user is present from a second space in which the virtual sound source is located, and a door provided on the wall are used,
The sound quality control method according to claim 6 , wherein the sensor is a sensor that detects whether the door is open or closed. The sensor detects the degree of opening and closing of the door,
The sound quality control method according to claim 7 , further comprising generating a sound quality obtained by cross-fading the first sound quality and the second sound quality in accordance with the degree of opening and closing of the door detected by the sensor. The sound quality control method according to claim 7 or 8, wherein the second sound quality is a sound quality filtered by an impulse response in the second space of the virtual sound source at the position of the open door and a head- related transfer function from the position of the open door to the user. The first sound quality is a sound quality filtered by an impulse response of the virtual sound source in the second space at the position of the open door, an impulse response of the sound insulation characteristic of the obstruction, and a head related transfer function from the position of the open door to the user;
The third sound quality is a sound quality filtered by a head-related transfer function from the localization position of the virtual sound source to the user.
The sound quality control method according to claim 9. A control unit of a mobile terminal device that communicates with an acoustic device worn by a user and a sensor that detects the movement of a movable obstruction,
generating a sound having a first sound quality when the sound source is blocked by the obstruction from a virtual sound source located on the opposite side of the obstruction, and emitting the sound from the acoustic device;
When the sensor detects that the obstruction has moved from a position obstructing the virtual sound source, the sound quality of the sound is changed from the first sound quality to a second sound quality when the virtual sound source is not obstructed by the obstruction ,
The sound quality control program causes the control unit to generate, when the user is in a position where the user can directly view the localization position of the virtual sound source, a sound of a third sound quality for when the user can directly view the localization position of the virtual sound source, instead of the second sound quality.
Sound quality control program .

Images