WO2018116368A1 - Playing sound provision device and recording medium - Google Patents

Abstract

In order that, even if, while performing a movement or the like relative to a player (10a) wearing a headphone (16a), another player (10c) plays a musical instrument, sense of incongruity given to the player (10a) wearing the headphone (16a) can be reduced, an azimuth angle (θ_ac) of a player (10b) is detected with reference to a direction (A_cntr) in which the head of the player (10a) faces, and audio signals based on the playing by the player (10c) are signal-processed, thereby localizing a sound image based on that playing to a direction (A_c) of the azimuth angle (θ_ac) with reference to the direction (A_cntr) and then outputting the localized sound image toward the headphone (16a) worn on the player (10a).

Description

Performance sound providing apparatus and recording medium

The present invention relates to, for example, a sound image providing apparatus and a recording medium that localize a sound image of a musical instrument played by another player as viewed from a certain player in the direction of the other player.

A technique is known in which when each of a plurality of performers plays an instrument, the sound image of each instrument is localized at the performance position of each instrument in a sound emission device (typically headphones) worn by each performer. (See Patent Document 1).

Patent No. 4426159

By the way, depending on the performance, the performer may perform while moving. In such a case, the position of the player who is visually recognized by the wearer of the sound emitting device does not match the sound image of the musical instrument played by the performer, which makes the wearer of the sound emitting device feel uncomfortable. It will end up.
The present invention has been made in view of such circumstances, and one of the purposes thereof is a feeling of strangeness given to the wearer of the sound emitting device even when the player plays the instrument while moving. It is to provide a technology for keeping the size small.

In order to achieve the above object, a performance sound providing apparatus according to an aspect of the present invention includes an azimuth angle detection unit that detects an azimuth angle of a second player based on a direction in which the head of the first player faces. The audio signal based on the performance by the second performer is signal-processed, and the sound image based on the performance is localized in the direction detected by the azimuth angle detection unit with respect to the direction in which the head of the first performer is facing. And a sound image processing unit that outputs the sound toward a sound emitting device attached to the first performer.

It is a block diagram which shows the structure of the performance sound provision apparatus which concerns on 1st Embodiment. It is a schematic diagram which shows the performance sound provision apparatus. It is a block diagram which shows the structure of the signal processing apparatus in the performance sound provision apparatus. It is a flowchart which shows operation | movement of the performance sound provision apparatus. It is a figure which shows operation | movement of the performance sound provision apparatus. It is a figure which shows operation | movement of the performance sound provision apparatus. It is a figure which shows operation | movement of the performance sound provision apparatus. It is a figure which shows operation | movement of the performance sound provision apparatus. It is a block diagram which shows the structure of the performance sound provision apparatus which concerns on 2nd Embodiment. It is a schematic diagram which shows the performance sound provision apparatus. It is a block diagram which shows the structure of the signal processing apparatus in the performance sound provision apparatus. It is a block diagram which shows the structure of the performance sound provision apparatus which concerns on a modification.

An outline of the performance sound providing apparatus 1 according to the embodiment of the present invention will be described.
In the performance sound providing apparatus 1, it is assumed that the four players perform a musical instrument and supply a stereo signal in which the sound image of each instrument is localized to each of the four player's headphones. However, this is not intended to limit the number of performers to “4”.

FIG. 5 is a diagram showing the performance positions of the four performers 10a, 10b, 10c, and 10d in a bird's-eye view (from a top view). Among the performers, the performer 10b performs the instrument 14b, the performer 10c performs the instrument 14c, and the performer 10d performs the instrument 14d, as shown in FIG. The performer 10a also plays a musical instrument, which is omitted in FIG.

Each of the musical instruments 14b, 14c, and 14d outputs performance results as audio signals. Examples of such musical instruments typically include electronic (electrical) musical instruments such as electronic pianos, electronic organs, and electric guitars. In addition, such an electronic musical instrument has a low volume, however, even if it is an acoustic musical instrument such as a grand (or upright) piano and a folk guitar, a sound collecting device such as a microphone If it is converted into an audio signal, the performance sound providing apparatus 1 can be used.

The performer 10a wears the headphones 16a. An autofocus camera 12a is mounted on the headband of the headphones 16a. The image photographed by the camera 12a is supplied to a signal processing device omitted in FIG. In the signal processing device, the image captured by the camera 12a is analyzed, and the azimuth angles of the other performers 10b, 10c, and 10d are detected with respect to the center direction Acntr where the head (face) of the performer 10a faces. Output.
Here, the azimuth angle is information indicating at what angle the target player is positioned in the left or right direction with respect to the direction Acntr in which a certain player's head faces. is there.

For example, when the performers 10a, 10b, 10c, and 10d are positioned as shown in FIG. 5 in plan view, the signal processing apparatus uses the direction Acntr as a reference as the azimuth angle of the performer 10b with respect to the performer 10a. Information indicating that the angle θab is in the left direction (counterclockwise in FIG. 5) is output, and the angle θac in the right direction (clockwise in FIG. 5) is set as the azimuth angle at which the player 10c is located with respect to the player 10a. Information indicating that the player 10d is positioned with respect to the player 10a is output as information indicating that the angle θad is in the right direction.

In addition, the signal processing device processes the audio signals output from each of the musical instruments 14b, 14c, and 14d, and applies a stereo signal in which the sound image is localized in the following direction to the headphones 16a worn by the performer 10a. Supply.
That is, the signal processing device localizes the sound image of the musical instrument 14b performed by the player 10b other than the player 10a in the direction Ab of the angle θab to the left with respect to the direction Acntr serving as the center for the player 10a. The sound image of the musical instrument 14c performed by the performer 10c is localized in the direction of the angle θac in the right direction, and the sound image of the instrument 14d performed by the performer 10d is localized in the direction of the angle θad in the right direction. Note that the signal processing device localizes the sound image of the musical instrument performed by the performer 10a itself, for example, within the head of the performer 10a.

Thereby, as shown in FIG. 6, from the player 10a wearing the headphones 16a, the sound image of the musical instrument 14b is localized in the direction in which the player 10b is located. Similarly, for the performer 10a, the sound image of the musical instrument 14c is localized in the direction of the performer 10c, and the sound image of the musical instrument 14c is localized in the direction of the performer 10c.
Then, by repeatedly performing such detection of the player's azimuth and localization of the sound image at predetermined time intervals, the player can perform while moving, or the head of the headphone wearer's head. Even if the performance is performed while changing the direction, the sound image of the musical instrument played by the performer is made to follow the direction of the performer, so that the performance sound with less sense of incongruity and full of realism is provided to the headphone wearer. be able to.

Here, the signal processing for the headphone 16a of the performer 10a has been described, but the same applies to the signal processing for the headphone 16b of the performer 10b, the headphone 16c of the performer 10c, and the headphone 16d of the performer 10d. is there.
The above is the outline of the performance sound providing apparatus 1. Next, details of the performance sound providing apparatus 1 will be described.

FIG. 1 is a diagram showing a configuration of a performance sound providing apparatus 1 according to the first embodiment, and FIG. 2 is a schematic diagram of the performance sound providing apparatus 1.

As shown in FIG. 1, the performance sound providing apparatus 1 includes a camera 12a, a musical instrument 14a, and headphones 16a corresponding to the performer 10a, a camera 12b, a musical instrument 14b, and headphones 16b corresponding to the performer 10b, and a performance. The camera 12c, the musical instrument 14c, and the headphones 16c corresponding to the player 10c, the camera 12d, the musical instrument 14d, and the headphones 16d corresponding to the performer 10d, and the signal processing device 20 are included.
It should be noted that each of the cameras 12a to 12d, the musical instruments 14a to 14d, and the headphones 16a to 16d is generally described without corresponding to a specific player, and a, b, c, and d at the end of the reference numerals are used. It explains without attaching. For example, the reference numeral is 16 for headphones when the performer is not specified.

In these drawings, a headphone 16a attached to the performer 10a emits a stereo signal processed by the signal processing device 20.
The camera 12a is a camera fixed to substantially the midpoint of the headband of the headphone 16a, and the shooting direction is the front side of the sheet in FIG. For this reason, the optical axis direction of the camera 12a substantially coincides with the direction in which the head of the performer 10a faces. Note that the angle of view of the camera 12a is preferably such that the other players 10b, 10c, and 10d can be photographed in the same field of view, for example, about 180 degrees.
If a situation is assumed in which other performers perform behind the performer 10a, the camera 12a may be an all-round camera.

The performer 10a plays the musical instrument 14a while wearing the headphones 16a. In the example of FIG. 2, the musical instrument 14 a is an electronic violin. The instrument 14 a converts a string vibration into, for example, a digital audio signal by a pickup disposed below the frame, and supplies the digital audio signal to the signal processing device 20.

The same applies to the camera 12b, the musical instrument 14b, and the headphones 16b corresponding to the performer 10b. The headphones 16b are attached to the performer 10b, and the camera 12b is fixed to almost the midpoint of the headband of the headphones 16b. The other player is photographed, and the musical instrument 14b is played by the player 10b.
Similarly, the headphone 16c is attached to the performer 10c, the camera 12c is fixed to substantially the midpoint of the headband of the headphone 16c, and the musical instrument 14c is played by the performer 10c. The headphone 16d is attached to the performer 10d, the camera 12d is fixed to substantially the midpoint of the headband of the headphone 16d, and the musical instrument 14d is played by the performer 10d.

In this example, the musical instrument 14b is an electric guitar, the musical instrument 14c is an electronic piano, and both the musical instruments 14b and 14c supply an audio signal to the signal processing device 20, for example, digitally. The musical instrument 14d is a saxophone that is an acoustic musical instrument, and the sound from the saxophone is converted into an audio signal by a sound collecting device inserted in a morning glory tube (bell), and is supplied to the signal processing device 20, for example, digitally.

FIG. 3 is a block diagram illustrating a configuration of the signal processing device 20. As shown in this figure, the signal processing device 20 includes a control unit 200, an input unit 210, a parameter generation unit 220, azimuth angle detection units 215a, 215b, 215c, and 215d, sound image processing units 230a, 230b, 230c and 230d.
Among these, the azimuth angle detection unit 215a and the sound image processing unit 230a correspond to the performer 10a, the azimuth angle detection unit 215b and the sound image processing unit 230b correspond to the performer 10b, and the azimuth angle detection unit 215c and the sound image processing unit 230c Corresponding to the player 10c, the azimuth angle detector 215d and the sound image processor 230d correspond to the player 10d.

The control unit 200, the azimuth angle detection units 215a, 215b, 215c, and 215d and the parameter generation unit 220 are functional blocks that are constructed when, for example, a CPU (Central Processing Unit) executes a predetermined program.
Note that the azimuth angle detectors 215a, 215b, 215c, and 215d are divided for convenience in order to be explained for each performer, but may be collectively considered as one azimuth angle detector. .

The input unit 210 is operated by one of the performers and supplies the operation content to the control unit 200. The control unit 200 receives content when the player 10 operates the input unit 210 and controls the azimuth angle detection units 215a, 215b, 215c, and 215d and the parameter generation unit 220.

The azimuth angle detector 215a analyzes the image taken by the camera 12a and detects the azimuth angles of the other players 10b, 10c, and 10d with respect to the player 10a.
As for the detection of such an azimuth angle, for example, first, the player 10b to be detected is specified by face recognition or the like, and second, when the camera 12a is focused on the face of the specified player 10b. Is obtained from the camera 12a, and thirdly, a distance Rab from the center of the photographed image (center direction Acntr) to the center of the face of the player 10a as the target is obtained. Third, the value of the arc sine obtained by dividing the distance Rab by the distance Lab may be calculated.
The azimuth angle detection unit 215a performs such detection while specifying other performers photographed by the camera 12a in order from the left to the right of the image, for example. Thereby, the azimuth angle detector 215a detects the azimuth angles of the other players 10b, 10c, and 10d with respect to the player 10a.

Similarly, the azimuth angle detection unit 215b detects the azimuth angles of the other players 10a, 10c, and 10d with respect to the player 10b, respectively, and the azimuth angle detection unit 215c detects the other players 10a, 10b, and 10d with respect to the player 10c. The azimuth angle is detected, and the azimuth angle detector 215d detects the azimuth angles of the other players 10a, 10b, and 10c with respect to the player 10d.

The parameter generation unit 220 generates various parameters such as sound image setting filter coefficients and gains based on the azimuth angle output from the azimuth angle detection unit 215a, and supplies the generated parameters to the sound image processing unit 230a. Similarly, the parameter generation unit 220 generates a sound image setting parameter based on the azimuth angle from the azimuth angle detection unit 215b (215c or 315d) and supplies it to the sound image processing unit 230b (230c or 230d).

The sound image processing units 230a, 230b, 230c, and 230d are a group of digital filters composed of one or a plurality of DSPs (Digital Signal Processors). Note that the sound image processing units 230a, 230c, 230c, and 230d are divided for convenience in order to be explained corresponding to each performer, but can be collectively considered as one sound image processing unit. .

The sound image processing unit 230a corresponding to the performer 10a performs a sound image localization process defined by parameters by the parameter generation unit 220 on each of the audio signals of the musical instruments 14a, 14b, 14c, and 14d, and performs a sound image localization process. Each stereo signal is mixed, converted into an analog signal, amplified, and supplied to the headphone 16a. In the sound image processing unit 230a, for example, the sound image of one musical instrument is localized using the technique described in Japanese Patent Laid-Open No. 10-23600. In this example, since the sound image processing unit 230a localizes the sound images of the four musical instruments, the four signals each localized using the above technique are added and supplied to the headphones 16a.

Briefly describing the above technique, the sound image processing unit 230a first delays the audio signal of the musical instrument 14b sequentially by a multi-stage delay circuit, and among these, the signal having the smaller delay time is used as the preceding sound (direct sound). ), And the signal having the longer delay time is extracted as the following sound (initial reflected sound). Secondly, the sound image processing unit 230a performs a convolution operation on the preceding sound and the following sound with, for example, FIR filters of the head-related transfer functions from eight directions to both ears, and performs the player on these FIR filters. The gain corresponding to the direction Ab of 10b is set, and the stereo signal of the musical instrument 14a is output.
The sound image processing unit 230a similarly processes the audio signals of the musical instruments 14c, 14d, and 14a, and outputs the stereo signal of the musical instrument 14c, the stereo signal of the musical instrument 14d, and the stereo signal of the musical instrument 14a, respectively. The sound image processing unit 230a adds these four stereo signals, converts them into analog signals, amplifies them, and supplies them to the headphones 16a.
As a result, the performer 10a wearing the headphones 16a localizes the sound images of the musical instruments 14a, 14b, 14c and 14d in the respective performance directions.

Similarly to the sound image processing unit 230a, the sound image processing unit 230b (230c or 230d) is defined by the filter coefficient generated by the parameter generation unit 220 for each of the audio signals of the musical instruments 14a, 14b, 14c, and 14d. The sound image localization process is performed, and the stereo signal subjected to the sound image localization process is supplied to the headphones 16b (16c or 16d).

FIG. 4 is a flowchart showing the operation of the sound image localization process in the signal processing apparatus 20 in the performance sound providing apparatus 1.
In order to generally describe the azimuth angle detection units 215a, 215b, 215c, and 215d and the sound image processing units 230a, 230b, 230b, and 230d, a, b, c, and d at the end of the code are used. May not be attached.

This sound image localization processing is executed in response to a predetermined operation, for example, an operation on the input unit 210.
First, in the signal processing device 20, the control unit 200 performs various initial settings (step Sa11). This initial setting includes, for example, a process of setting the number of performers and a process of registering the correspondence between the performer 10, the musical instrument 14, and the headphones 16.
As for the number of performers, for example, the number of musical instruments and the number of sets of headphones connected to the signal processing device 20 may be detected, or any performer 10 may input to the input unit 210. The number of performers may be obtained by analyzing (face recognition) an image captured by a certain camera 12 and adding “1” to the number obtained by the analysis. Here, the reason why “1” is added to the number of persons obtained by analysis is that the player 10 wearing the headphones 16 becomes a blind spot of the camera 12 mounted on the headphones 16.
The reason why the correspondence relationship between the performer 10, the musical instrument 14, and the headphones 16 needs to be registered is that, for example, when the performer 10 is identified by face recognition from the photographed image, the identified performer 10 determines which instrument 14 This is because it is necessary to grasp which headphones 16 are worn.

Next, the control unit 200 selects any one of the initially set number of players, that is, pays attention to one player (step Sa12). Here, the reason for paying attention to one performer is to select and process the performers one by one in order when supplying the signals in which the sound images are localized to the headphones of all performers. .

Subsequently, the control unit 200 specifies one player other than the player of interest (step Sa13). The reason for identifying one of the other performers is that the other performers are processed one by one in order to localize the sound image of the instrument performed by other performers other than the performer of interest. Because.

The control unit 200 causes the azimuth angle detection unit 215 corresponding to the focused player to detect the azimuth angle of the specific player from the image captured by the camera 12 mounted on the focused player's headphones 16. . Thereby, the azimuth angle detection unit 215 detects the azimuth angle and supplies the azimuth angle to the parameter generation unit 220, and the parameter generation unit 220 acquires the azimuth angle (step Sa14).
When the azimuth angle of the specific player with respect to the player of interest is acquired, the control unit 200 causes the sound image processing unit 230 corresponding to the player of interest to obtain a sound image of the musical instrument played by the specific player with the acquired azimuth angle. The direction is localized (step Sa15). Specifically, the control unit 200 causes the parameter generation unit 220 to cause the sound image processing unit 230 corresponding to the player of interest to localize the sound image of the musical instrument played by the specific player in the direction of the acquired azimuth angle. Instructing the supply of the parameter, the sound image processing unit 230 performs a sound image localization process on the audio signal of the musical instrument according to the parameter.

The control unit 200 determines whether or not the sound image localization processing has been executed for all other players other than the player of interest (step Sa16).
If the determination result is “No”, the control unit 200 changes the other player to be identified to a person who has not performed the sound image localization processing (step Sa17), and then returns the processing procedure to step Sa14. In step Sa14 and Sb15, the sound image localization processing of the musical instrument played by the other changed player is executed.

On the other hand, if the determination result in step Sa16 is “Yes”, that is, if the sound image localization processing is performed for each instrument played by a player other than the target player, the control unit 200 performs the target performance. The sound image localization process of the musical instrument performed by the player is executed, and the sound image of the musical instrument is localized, for example, in the head of the focused player (step Sa18).
When step Sa18 is completed, the sound images of all musical instruments are localized with respect to the headphones 16 of the player 10 of interest. Therefore, the control unit 200 next determines whether or not all the performers have been paid attention to in order (step Sa19).
If this determination result is “No”, the control unit 200 changes the focused player to one of the players not focused (step Sa20), and then returns the processing procedure to step Sa12. Then, the processing of steps Sa14 to Sa18 is executed corresponding to the changed focused player.

If the determination result in step Sa19 is “Yes”, that is, if attention is paid to all the performers, the control unit 200 determines, for example, the amount of change between the previously acquired azimuth angle and the currently acquired azimuth angle as the focused player. Is determined for all of the specific performers, and it is determined whether or not the average value is equal to or less than a predetermined threshold value (step Sa21).

If the determination result is “Yes”, that is, if the relative movement of each player 10 is relatively small, the control unit 200 increases the time interval for executing the sound image localization processing by one step (step Sa22).
On the other hand, if the determination result is “No”, that is, if the relative movement of each player 10 is relatively large, the control unit 200 further shortens the time interval for executing the sound image localization processing (step Sa23).

After the process of step Sa22 or Sa23, the control unit 200 determines whether or not the changed time interval has elapsed since the previous execution (step Sa24).
If it has not elapsed (if the determination result in step Sa24 is “No”), the control unit 200 returns the processing procedure to step Sa24. That is, the control unit 200 stands by in step Sa24 until the changed time interval elapses.
On the other hand, if the changed time interval has elapsed since the previous execution (if the determination result in step Sa24 is “Yes”), the control unit 200 returns the processing procedure to step Sa12.
As a result, steps Sa12 to Sa24 are executed again, and the other player performs the performance in the direction in which the other player plays on the player's headphone 16 with reference to the direction of the head of the player. The process of localizing the sound image of the musical instrument to be performed is executed corresponding to each performer.

For example, when four performers 10a, 10b, 10c, and 10d perform at the position shown in FIG. 5 described above, as shown in FIG. 6, the sound image of the instrument 14b is displayed on the headphone 16a. The sound image of the musical instrument 14c is localized in the direction Ac of the performer 10c, and the sound image of the musical instrument 14d is localized in the direction Ad of the performer 10b. For this reason, the player 10a wearing the headphones 16a can experience a sense of reality as if other players are playing at that point.

Further, among the four performers, for example, when the performer 10c moves from the position of the broken line shown in FIG. 7 to the position of the solid line toward the direction Acntr (left side in the drawing), the player 10c is selected in the headphones 16a. The sound image of the musical instrument 14c to be played changes with the movement. Specifically, in the sound image localization processing of FIG. 4, when the focused player is the player 10a (step Sa12) and the specific player is the player 10c (step Sa13 or Sa17), the head of the player 10a. The azimuth angle detection unit 215a detects that the player 10c is positioned on the left side in the direction Ac 'of the angle θac' with respect to the direction Acntr to which the angle is directed, and the detected azimuth angle is acquired by the parameter generation unit 220. (Step Sa14). As a result, the sound image of the musical instrument 14b is localized in the direction Ac ′ by the sound image processing unit 230a (step Sa15).

The movement of the player 10c is processed in the same manner when the player of interest becomes the player 10b (10d), and the sound image of the instrument 14c is localized at the point after the movement in the headphones 16b (16d).
On the other hand, when the performer 10c moves, even if the other performers 10a, 10b, and 10d do not actually move, the player 10c moves relatively. For this reason, when the focused player becomes the player 10c, the sound images of the musical instruments 14a, 14b, and 14d are localized at the position after the relative movement in the headphones 16c.

If the average value of the change amount from the direction Ac to Ac ′ of the player 10c viewed from the player 10a and the other change amount exceeds the threshold value, it is determined that the movement is relatively large, and steps Sa12 to Sa24 are performed. The time interval for executing is changed by one step. For this reason, since the sound image localization is frequently executed at short intervals, the sound image can be localized while following the movement.
On the other hand, if the average value is less than or equal to the threshold value, it is determined that the movement is relatively small, and the time interval for executing steps Sa12 to Sa24 is changed by one step (step Sa22).

Among the four performers, for example, when only the performer 10a points his head in the direction Acntr '' shown in FIG. 8 from the direction Acntr shown in FIG. 5, the other performers 10b, 10c, and 10d Even if they are not actually moving, they are moving relative to the player 10a. For this reason, when the focused performer becomes the performer 10a, the sound images of the musical instruments 14b, 14c, and 14d are localized in the direction after relative movement in the headphones 16a.
In detail, the sound image of the musical instrument 14b is localized in the direction Ab ″ of the angle θab ″ to the left with respect to the direction Acntr toward the head of the performer 10a, and the sound image of the musical instrument 14c is based on the direction Acntr. The sound image of the musical instrument 14d is localized in the direction Ad ″ at the angle θad ″ with respect to the direction Acntr.

As described above, in the performance sound providing apparatus 1, when another player moves relative to a certain target player, the other player performs the movement with the headphone 16 worn by the target player. The sound image of the musical instrument 14 to be played also moves. Therefore, even if another player plays the instrument while moving, or the other player's position moves relative to the head due to a change in head direction (center direction), Since the sound image of the musical instrument played by the performer follows the movement of the performer, it is possible to provide a performer wearing headphones with less sense of incongruity and full of realism.

By the way, in the performance sound providing apparatus 1 according to the first embodiment, the azimuth angle of the specific player with respect to the player of interest is detected by analyzing the image photographed by the camera 12, but other than this. The configuration can also be detected. Next, a second embodiment in which the azimuth angle is detected with another configuration will be described.

FIG. 9 is a diagram showing a configuration of the performance sound providing apparatus 1 according to the second embodiment, and FIG. 10 is a schematic diagram of the performance sound providing apparatus 1.

The second embodiment is different from the first embodiment in that a camera 13 is provided instead of the camera 12. The camera 13 is an omnidirectional camera that is installed on the ceiling or the like and shoots all directions (360 degrees) from the installation point, that is, shoots all performers at once at the installation point.
In addition, the image image | photographed with the camera 13 is supplied to the signal processing apparatus 20 similarly to 1st Embodiment.

FIG. 11 is a block diagram illustrating a configuration of the signal processing device 20 in the performance sound providing device 1 according to the second embodiment.
As shown in this figure, in the signal processing device 20 in the second embodiment, an azimuth angle detector 217 is provided corresponding to the camera 13.

In the second embodiment, when the camera 13 is installed on the ceiling, all the performers 10 can be photographed at once. However, since only a part of the player 10, particularly the head, may be photographed, the player 10 cannot be identified by face recognition, and the direction in which the player 10's head faces cannot be detected. It is also assumed.

Therefore, in the second embodiment, the azimuth angle detection unit 217 identifies the performer 10 by, for example, the following image processing, and detects the position of the performer and the direction in which the performer's 10 head faces.
First, as a preliminary preparation, earphones for both ears of the headphone 16 to be worn by the performer 10 are each provided with a marker that can identify the left and right for each headphone 16 when image processing is performed.

The azimuth angle detection unit 217 analyzes the image taken by the camera 13 installed on the ceiling, identifies the headphones from the markers attached to the ear cups of both ears, and identifies the performer wearing the headphones. At the same time, the midpoint of the straight line connecting the ear cups of both ears is detected as the position of the performer, and the direction substantially orthogonal to the straight line is detected as the direction in which the head (face) of the performer faces.

Specifically, for example, in FIG. 5, when the heading 16a is identified from the marker, the azimuth angle detection unit 217 uses the midpoint of the straight line connecting the ear cups of both ears as the position of the player 10a wearing the headphone 16a. If the left-ear ear cup (L) is at 9 o'clock and the right-ear ear cup (R) is at 3 o'clock with reference to the midpoint of the straight line, the 12 o'clock direction is Then, the direction Acntr facing the head of the player 10a is detected.
Similarly, the azimuth angle detection unit 217 detects the position and the direction of the head, respectively, for the performers 10b, 10c, and 10d.

If the position of each player 10a, 10b, 10c, and 10d and the direction in which the head faces are known, the azimuth angle detection unit 217 uses geometric calculations to determine the performance of other players when viewed from one player. Each azimuth can be determined.
In the first embodiment, a stereo signal obtained by localizing the sound image of the musical instrument played by the other player at the azimuth angle of the other player is supplied to the headphones worn by the one player. It is the same.

According to the second embodiment, the player 10 can experience a sense of realism as if he / she is performing while moving, and the camera 12 does not have to be mounted on the headphones 16 according to the second embodiment. The burden on the player 10 can be reduced.

The present invention is not limited to the first embodiment and the second embodiment described above, and various applications and modifications as described below, for example, are possible. Note that one or a plurality of arbitrarily selected aspects of application / deformation described below can be appropriately combined.

In the first embodiment and the second embodiment, the signal processing device 20 is shared over the performers 10a, 10b, 10c, and 10d, but a signal processing device may be prepared for each performer.
FIG. 12 shows a signal processing device 20a corresponding to the performer 10a, a signal processing device 20b corresponding to the performer 10b, a signal processing device 20c corresponding to the performer 10c, and a performer 10d. In this example, signal processing devices 20d are provided.
Note that the signal processing device 20a corresponding to the performer 10a detects the azimuths of the performers 10b, 10c, and 10d with respect to the performer 10a from the image captured by the camera 12a, and the performers 10b, 10c, and 10d. The sound image of the musical instrument played by the player is localized in the direction of the detected azimuth angle and supplied to the headphones 16a worn by the performer 10a.
The signal processing device 20b corresponding to the performer 10b detects the azimuths of the performers 10a, 10c, and 10d with respect to the performer 10b from the image captured by the camera 12b, and the performers 10a, 10c, and 10d The sound image of the musical instrument to be played is localized in the direction of the detected azimuth angle and supplied to the headphones 16b worn by the performer 10b.
The same applies to the signal processing device 20c corresponding to the performer 10c and the signal processing device 20d corresponding to the performer 10d.

In the first embodiment and the second embodiment, the sound image of the musical instrument played by the other player is localized in the direction of the azimuth angle of the other player as viewed from a certain player.
Here, a sensor for detecting the direction of the head of a certain player may be provided. For example, with respect to the azimuth angle of another player, this sensor can measure at any angle in either the left or right direction when a player points his head from the initial position (reference position) to the other player. Detect if there was. Then, when the performance is returned to the initial position, the sound image of the musical instrument played by another player may be localized in the direction of the detected azimuth angle.

The sound image to be given to a certain player's headphones may be configured to be localized in a direction in which the other player is located and a distance from the certain player to the other player. That is, the sound image may be localized at a three-dimensional point in consideration of not only the direction but also the distance. In addition, since the sound image of the musical instrument played by the performer is more precisely the point of the instrument rather than the performer, the instrument is recognized by pattern matching, etc., and is localized in the direction or point where the instrument is recognized. You may let them.
In addition, the musical instrument played by the performer wearing the headphones may be localized at the point of the musical instrument instead of in the head.

Assuming that a performer is performing in a virtual space prepared in advance, a sound field in that space is given, and the sound image of the instrument played by another performer is displayed on the performer's headphones. You may localize to the point.
Note that the virtual space must be prepared in advance, such as halls, churches, and studios, with multiple sizes of space and intensity of reflected sound. As well as how the performer is to be placed in the selected space.

The sound image to be localized is not limited to musical instruments, and may be a singer. That is, the performance includes singing, and the sound represented by the audio signal includes singing voice.

From the above-mentioned embodiments and application / deformation modes, the following inventions can be grasped from the viewpoint of minimizing the uncomfortable feeling given to the headphone wearer even when the performer plays the instrument while moving. Is done.

First, an azimuth angle detection unit that detects the azimuth angle of the second player with respect to the direction in which the head of the first player is facing, and an audio signal based on the performance by the second player are signal-processed, and the performance Is localized in a direction detected by the azimuth angle detection unit with reference to a direction in which the head of the first player faces, and is output toward a sound emitting device attached to the first player. A performance sound providing apparatus including a sound image processing unit is grasped. According to the performance sound providing apparatus, even when the performer plays a musical instrument while moving or the like, the uncomfortable feeling given to the wearer of the sound emitting device such as headphones can be suppressed to a small level.
Note that “towards” means that another intermediate element such as an amplifier circuit may be interposed in the middle.

In the performance sound providing apparatus, the azimuth angle detection unit analyzes an image photographed by a photographing device collocated on the head of the first performer, and the second performer with respect to the first performer. The configuration for detecting the azimuth angle of is determined. According to the above configuration, it is possible to detect the azimuth angle of the second player with respect to the first player without directly obtaining the direction toward the head of the first player.

In the performance sound providing apparatus, the azimuth angle detecting unit analyzes an image obtained by photographing the first player and the second player by a photographing device installed at a predetermined point, and provides the first player with the first sound. A configuration for detecting the azimuth angle of the second player is grasped. According to the above configuration, it is possible to reduce the burden on the first performer when detecting the azimuth angle of the second performer with respect to the first performer.

In the performance sound providing apparatus, it is understood that the azimuth angle acquisition by the direction acquisition unit and the sound image localization by the sound image processing unit are executed at predetermined time intervals. According to the above configuration, the sound image can appropriately follow the change in the orientation of the head of the first player and / or the movement of the second player.

In the performance sound providing apparatus, the azimuth angle obtaining unit obtains a distance from the first player to the second player, and the sound image processing unit obtains a sound image based on the performance by the second player, A configuration in which localization is performed at a point of the direction and distance acquired by the azimuth acquisition unit with respect to the direction in which the head of the first player faces is grasped. According to the above configuration, the accuracy of localization of the sound image can be improved.

In the performance sound providing apparatus, the azimuth angle detection unit detects an azimuth angle of the first player with reference to a direction in which the head of the second player is facing, and the sound image processing unit is configured to detect the first player. The audio signal based on the performance by the signal is signal-processed, and the sound image based on the performance is localized in the direction detected by the azimuth angle detection unit with respect to the direction in which the head of the second player faces. A configuration for output to a sound emitting device attached to the performer is grasped. According to the said structure, a 1st player and a 2nd player can be made to experience an ensemble.

Further, the computer is based on the performance by the second player with respect to the azimuth angle detecting unit for detecting the azimuth angle of the second player with respect to the direction in which the head of the first player faces, and the sound image processing unit. The audio signal is subjected to signal processing, and a sound image based on the performance is localized in a direction acquired by the azimuth acquisition unit with reference to a direction in which the head of the first player faces, and is attached to the first player. A recording medium on which a computer program that functions as a control unit that controls to output the sound to the sound emitting device is recorded is grasped. According to the recording medium described above, when the computer functions as a performance sound, even if the performer plays the instrument while moving, the uncomfortable feeling given to the wearer of the sound emitting device such as the headphones can be suppressed. it can.

DESCRIPTION OF SYMBOLS 1 ... Performance sound provision apparatus, 10a, 10b, 10c, 10d ... Performer, 12a, 12b, 12c, 12d ... Camera, 14a, 14b, 14c, 14d ... Musical instrument, 16a, 16b, 16c, 16d ... Headphone, 20 ... Signal processing device, 200 ... control unit, 215a, 215b, 215c, 215d ... azimuth angle detection unit, 230a, 230b, 230c, 230d ... sound image processing unit.

Claims (6)

1. An azimuth angle detection unit for detecting the azimuth angle of the second player with reference to the direction in which the head of the first player faces;
   An audio signal based on the performance by the second player is signal-processed, and a sound image based on the performance is localized in a direction detected by the azimuth angle detection unit with respect to a direction in which the head of the first player faces. A sound image processing unit that outputs the sound to a sound emitting device attached to the first performer;
   A device for providing performance sounds.
2. The azimuth angle detector
   The performance sound providing apparatus according to claim 1, wherein an image taken by a photographing device mounted on the head of the first performer is analyzed to detect an azimuth angle of the second performer with respect to the first performer. .
3. The azimuth angle detector
   The imaging device installed at a predetermined point analyzes an image of the first player and the second player and detects an azimuth angle of the second player with respect to the first player. Performance sound providing device.
4. The performance sound providing apparatus according to claim 1, wherein acquisition of an azimuth angle by the direction acquisition unit and localization of a sound image by the sound image processing unit are performed at predetermined time intervals.
5. The azimuth angle detector
   Detecting the azimuth angle of the first player based on the direction in which the head of the second player is facing;
   The sound image processing unit
   An audio signal based on the performance by the first player is signal-processed, and a sound image based on the performance is localized in a direction detected by the azimuth angle detection unit with respect to a direction in which the head of the second player faces. The performance sound providing apparatus according to claim 1, wherein the performance sound is output toward a sound emitting device attached to the second performer.
6. Computer
   An azimuth angle detection unit that detects the azimuth angle of the second performer with reference to the direction in which the head of the first performer is facing; and
   An audio signal based on the performance by the second player is signal-processed with respect to the sound image processing unit, and a sound image based on the performance is acquired by the azimuth acquiring unit with reference to a direction in which the head of the first player faces. A control unit for controlling the sound to be output toward a sound emitting device attached to the first player,
   A recording medium that records a computer program that functions as a computer.

Images