JP2003284196A - Sound image localizing signal processing apparatus and sound image localizing signal processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a sound image localizing signal processing apparatus and a sound image localizing signal processing method capable of localizing a sound image in a direction of a moving video image even in the case of moving and reproducing the video image with a changed angle for reproduction of an audio signal with respect to a free viewpoint video image. <P>SOLUTION: The sound image localizing signal processing apparatus includes: a viewpoint selection section 5 for selecting viewpoint selection information of a user acting like angle information of a reproduced sound image of a video image on the basis of a video signal; a channel mapping section 6 for applying signal processing in real time for localizing the sound image to a channel of the audio signal corresponding to the viewpoint selection information of the user used for angle information of the selected reproduction sound image; a meta data section 9 for producing meta data to perform the signal processing; and an audio interleave section 11 for superimposing the audio signal and the meta data on the video signal to produce transmission information, and applies signal processing to the audio signal in real time so as to control the sound image localizing position corresponding to the video image on the basis of the video signal. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sound image localization signal processing device and a sound image localization signal processing method for performing virtual sound source localization processing on an audio signal corresponding to a video signal having angle information.

[0002]

2. Description of the Related Art Conventionally, there has been known an omnidirectional camera that subjects a surrounding object of 360 degrees to, for example, eight view angles and continuously processes the images to pick up images. The 360-degree imaged image captured by this omnidirectional camera can be displayed without paying attention to the viewpoint.
It was called a free-viewpoint video because it captured the surroundings at 0 degrees. The sound corresponding to this free-viewpoint image was picked up via a microphone so that a stereo sound field could be reproduced by fixing it to an image of a 360 ° surrounding subject divided into, for example, 8 field angles.

[0003]

As described above, in reproducing the audio signal for the above-mentioned conventional free-viewpoint video, even if the output is a stereo audio output signal, such a stereo audio output signal is reproduced. In this case, the normal sound image is localized only between the two speakers used as the reproducing means, and is not localized in the other directions.

Therefore, when an image reproduced on a monitor such as a television receiver at the time of reproduction is reproduced by moving by changing the angle of the 360-degree imaged image, the sound image is reproduced as 3.
For example, the sound image does not match the moving image displayed on the monitor, and is fixed to the image of the surrounding object of 60 degrees divided into, for example, 8 view angles, and the normal sound image is reproduced. There is an inconvenience that the sound is localized only between the speakers and not in the other directions. In the game machine, the sound source and the image are stored in the memory as data,
There are some that change the position of the sound source with respect to the image read from the memory, but there is a demand for technology that changes the position of the sound source in real time with respect to the image read in a stream in real time, such as video signals and audio signals. It was

Therefore, the present invention has been made in view of the above point, and in reproducing an audio signal for a free-viewpoint image, the direction of the moving image is changed even when the image is moved by changing the angle and reproduced. An object is to provide a sound image localization signal processing device and a sound image localization signal processing method capable of localizing a sound image.

[0006]

A sound image localization signal processing device of the present invention localizes a reproduced sound image at the time of reproduction so as to correspond to a video image based on a video signal having angle information with respect to a reference position. In a sound image localization signal processing device that performs signal processing on an audio signal obtained by synthesizing sound source data input from a plurality of sound sources during production as described above, a user who becomes angle information of a reproduced sound image of a video based on the video signal. Selection means for selecting the viewpoint selection information, and signal processing for sound image localization is performed in real time on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected by the selection means. Sound image localization control processing means and control information for generating control information for performing signal processing by the sound image localization control processing means. The audio signal and the control information are superposed on the video signal to generate transmission information, and the transmission information generation means is provided, and the audio signal is subjected to signal processing in real time and is based on the video signal. The sound image localization position is controlled according to the image.

Further, the sound image localization signal processing device of the present invention is
An audio signal obtained by synthesizing sound source data input from a plurality of sound sources by a synthesizing unit and performing signal processing according to control information at the time of production so as to correspond to a video based on a video signal having angle information with respect to a reference position. In the sound image localization signal processing device that performs signal processing on the reproduction information so as to localize the reproduced sound image at the time of reproduction, from the audio signal and the control information superimposed on the video signal in the reproduction information, Reproduction information reading means for reading the video signal, the audio signal, and the control information, a video signal reproducing means for reproducing the video signal, and a user's viewpoint selection as angle information of a reproduced sound image of the video based on the video signal. Selecting means for selecting information, and a user which becomes angle information of the reproduced sound image selected by the selecting means. Sound image localization control processing means for performing signal processing for sound image localization in real time on the channel of the audio signal corresponding to the viewpoint selection information of the audio signal, and performing signal processing on the audio signal in real time, based on the video signal. The sound image localization position is controlled according to the image.

Further, the sound image localization signal processing method of the present invention is
By synthesizing each sound source data input from a plurality of sound sources at the time of production so as to localize the reproduced sound image at the time of reproduction so as to correspond to the video based on the video signal having the angle information with respect to the reference position In a sound image localization signal processing method for subjecting a synthesized audio signal to signal processing, a selection step of selecting user's viewpoint selection information to be angle information of a reproduced sound image of a video based on the video signal, and a reproduction selected by the selection step. In order to perform signal processing by the sound image localization control processing step that performs signal processing for sound image localization in real time on the channel of the audio signal that corresponds to the user's viewpoint selection information that is the angle information of the sound image, and the sound image localization control processing step Control information generating step for generating control information of the A transmission information generating step of generating a transmission information by superimposing a signal and the control information, and subjecting the audio signal to signal processing in real time to control the sound image localization position corresponding to the video based on the video signal. It is something that is done.

The sound image localization signal processing method of the present invention is
An audio signal obtained by synthesizing sound source data input from a plurality of sound sources by a synthesizing unit and performing signal processing according to control information at the time of production so as to correspond to a video based on a video signal having angle information with respect to a reference position. In the sound image localization signal processing method for performing signal processing on the reproduction information so as to localize the reproduction sound image at the time of reproduction, in the audio signal and the control information superimposed on the video signal in the reproduction information, A reproduction information reading step for reading the video signal, the audio signal, and the control information, a video signal reproducing step for reproducing the video signal, and a user's viewpoint selection which becomes angle information of a reproduced sound image of a video based on the video signal. The selection step for selecting information and the angle of the reproduced sound image selected in the above selection step A sound image localization control processing step of performing signal processing for sound image localization in real time on a channel of the audio signal corresponding to the user's viewpoint selection information to be information, and performing the signal processing on the audio signal in real time; The sound image localization position is controlled according to the image based on the signal.

Therefore, according to the present invention, the following operations are performed. In the production system, the sound image localization control processing means performs signal processing for sound image localization in real time on the channel of the audio signal based on the control information corresponding to the angle information of the image from the selection means.

Further, the control information generating means uses the control information used in the sound image localization processing for performing signal processing for sound image localization on the channel of the audio signal corresponding to the angle information of the image by the sound image localization control processing means. Store.

The sending information generating means uses the audio signal and the control signal for the video signal to generate sending data based on the sending format. This transmission data is transferred to the reproduction system.

Thus, in the production system, the control information used for the real-time sound image localization processing for performing the signal processing for sound image localization on any channel of the audio signal is generated, and the video signal of the free viewpoint video and By transferring to the reproduction system together with the audio signal, it is possible to localize the reproduced sound image of the audio signal in the direction of the moving image when the image is moved and reproduced by changing the angle of the free viewpoint image.

In the reproduction system, the sound image localization control processing means performs signal processing for sound image localization on the channel of the audio signal transferred from the production system based on the control information corresponding to the angle information of the image from the selection means. Apply in real time.

The corrected audio signal, which has been subjected to the signal processing for sound image localization on the channel of the audio signal based on the control information corresponding to the angle information of the image by the sound image localization control processing means, is output to the reproduction means. The reproducing means reproduces the corrected audio signal and outputs a reproduced sound image corresponding to the angle information of the image.

As a result, in the reproduction system, the control information used for the sound image localization processing for performing the signal processing for the sound image localization on an arbitrary channel of the audio signal and outputting the corrected audio signal is used to generate the free viewpoint video. By performing sound image localization processing on the audio signal for the video signal of, the playback sound image of the audio signal can be localized in the direction of the moving image when the image is moved by changing the angle of the free-viewpoint image. it can.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION The sound image localization signal processing apparatus applied to this embodiment is configured so that the sound image localization position changes as the user changes the viewpoint in reproducing the sound field of the audio signal in the so-called free viewpoint video. In order to achieve this, the sound creator transmits control information for localizing the reproduced sound image of the audio signal in accordance with the video when producing the video content, as metadata of the authoring information to the reproduction side.

A sound image localization signal processing device applied to this embodiment will be described. FIG. 1 is a block diagram showing a configuration of a sound image localization signal processing device of a production system of a free-viewpoint video / audio production / reproduction system. The sound image localization signal processing device shown in FIG. 1 controls an audio signal so as to localize a reproduced sound image of an audio signal in a moving image direction when the image is moved and reproduced by changing an angle of a free viewpoint image. The point that the metadata is transmitted to the reproducing side is a big difference from the conventional one.

In FIG. 1, in the production system, the microphone material 1 of the free-viewpoint camera and other materials 2 are input to the mixer 3 as audio input of a plurality of channels. The microphone material 1 of the free-viewpoint camera is a sound corresponding to the free-viewpoint video, and it is fixed to the video of the surrounding subject of 360 degrees divided into, for example, 8 angles of view, and a stereo sound field is reproduced via a microphone. Is picked up. Other materials 2 are
It is called a dry material, and is a spot microphone material, a post-record microphone material or a sound effect material.

In the mixer 3, the sound creator uses the free viewpoint viewer 4 to input the audio input from a plurality of channels of sound sources such as the microphone material 1 of the free viewpoint camera and other materials 2 to the free viewpoint video of the video data 10 by 360 degrees. It is a sound adjustment table that allows you to adjust the composition processing for each channel while watching the material of the image by looking at it over. For example, N channel (for example, 24 channels) audio input data is combined with M (<N) channel (for example, 4 to 10 channels) audio output data and adjusted.

In the free viewpoint viewer 4, the angle information of the free viewpoint video is selected by operating the operation key for selecting the viewpoint in the viewpoint selecting unit 5. The operation key for selecting a viewpoint in the viewpoint selecting unit 5 is, for example, an operation for selecting angle information by rotating the operation key and a determination operation for pressing the operation key.

The mixer 3 is provided with a channel mapping section 6 as a functional block. The audio output data from the mixer 3 is supplied to the channel mapping unit 6. The angle information of the free viewpoint video is supplied from the viewpoint selecting unit 5 to the channel mapping unit 6. Therefore,
The channel mapping unit 6 performs signal processing for sound image localization on the channel of audio output data from the mixer 3 in real time based on the control information corresponding to the angle information of the free viewpoint video from the viewpoint selection unit 5. The channel mapping unit 6 inputs, for example, audio data of synthesized sound sources of 4 to 10 channels and outputs multi-audio data of 2 channels to 5.1 channels, 6 channels, 7 channels, and the like. The details of the channel mapping unit 6 will be described later.

The corrected audio data, which has been subjected to signal processing for sound image localization in the channel of the audio data based on the control information corresponding to the angle information of the free viewpoint video in the channel mapping unit 6, is output to the audio monitor 7. The audio monitor 7 reproduces the corrected audio data and outputs a reproduced sound image corresponding to the angle information of the free viewpoint video. The audio monitor 7 includes a speaker for reproducing left (L) and right (R) stereo sound, a left (L),
Headphones that play right (R) stereo sound, left (L), right (R), surround left (SL), speakers that play stereo sound of four channels of surround right (SR) or left (L), right A speaker that reproduces 5.1-channel stereo sound of (R), center (C), surround left (SL), surround right (SR), and subwoofer (SW) may be used.

Further, the control information used in the sound image localization processing of the corrected audio data in which the signal processing for the sound image localization is performed on the channels of the audio data corresponding to the angle information of the free viewpoint video in the channel mapping unit 6 is: It is supplied to the metadata section 9 as metadata of authoring information.

The audio output data from the mixer 3 is
The audio recorder 8 is supplied to the audio recorder 8 and streams the audio data into a format for transmission in synchronization with, for example, a synchronization signal from the channel mapping unit 6.

The audio interleave unit 11 interleaves the audio data streamed into the video data 10 and the metadata from the metadata unit 9 and sends the sending data 1 based on the sending format.
Generates 2. The transmission data 12 is transferred to a reproduction system described later via the Internet 13 or the recording medium 14, for example.

With this, in the production system, the control information used for the real-time sound image localization processing for performing the signal processing for sound image localization on an arbitrary channel of the audio data and outputting the corrected audio data, By transferring to the playback system together with the video data and audio data of the free-viewpoint video, if the angle of the free-viewpoint video is changed and the video is played back, the reproduced sound image of the audio signal is localized in the direction of the moving video. be able to.

FIG. 2 is a block diagram showing the configuration of the sound image localization signal processing device of the reproduction system of the free-viewpoint video / audio production / reproduction system. 2, in the reproducing system, the transmission data 12 transferred from the above-described production system shown in FIG. 1 via the Internet 13 or the recording medium 14, for example.
Is input as the reproduction data 21. Video data, audio data, and metadata are read from the reproduction data 21 based on the transmission format, the video data is supplied to the video player 22, and the audio data is a channel provided in an audio player (not shown). The metadata supplied to the mapping unit 24 and provided in the audio player (not shown) is supplied to the metadata unit 25.

In the video player 22, the viewpoint selecting section 23
The angle information of the free viewpoint image is selected by operating the operation key for selecting the viewpoint in. The free viewpoint video reproduced by the video player 22 is reproduced based on the angle information selected by the viewpoint selection unit 23. The operation key for selecting a viewpoint in the viewpoint selection unit 23 is, for example, an operation for selecting angle information by rotating the operation key and a determination operation for pressing the operation key.

The channel mapping unit 24 is supplied with angle information of a free viewpoint video from the viewpoint selecting unit 23 and metadata as control information from the metadata unit 25. Therefore, the channel mapping unit 24 performs signal processing for sound image localization on the channel of the audio data transferred from the production system based on the metadata as the control information corresponding to the angle information of the free viewpoint video from the viewpoint selection unit 23. Apply in real time. Channel mapping unit 24
Inputs, for example, audio data of synthesized sound sources of 4 to 10 channels and outputs multi-audio data of 2 to 5.1 channels, 6 channels, 7 channels, and the like. The details of the channel mapping unit 24 will be described later.

The corrected audio data, which has been subjected to the signal processing for sound image localization on the channel of the audio data based on the metadata as the control information corresponding to the angle information of the free viewpoint video in the channel mapping unit 24, is displayed on the audio monitor 27. Is output. The audio monitor 27 reproduces the corrected audio data and outputs a reproduced sound image corresponding to the angle information of the free viewpoint video. The audio monitor 27 includes a speaker for reproducing left (L) and right (R) stereo sound, headphones for reproducing left (L) and right (R) stereo sound, and left (L) and right (R). , Surround left (SL), surround right (SR) 4-channel speaker or left (L), right (R), center (C), surround left (SL), surround right (SR), A speaker that reproduces 5.1-channel stereo sound of a subwoofer (SW) may be used.

As a result, in the reproduction system, the control information used for the sound image localization processing for performing the signal processing for the sound image localization on an arbitrary channel of the audio data and outputting the corrected audio data is used for the free viewpoint video. By performing sound image localization processing on the audio data for the video data of, the playback sound image of the audio signal can be localized in the direction of the moving image when the image is moved and reproduced by changing the angle of the free-viewpoint image. it can.

The details of the channel mapping section will be described below. FIG. 3 is a block diagram showing the configuration of the channel mapping unit. In FIG. 3, the channel mapping unit uses an operation mode setting unit 31 that sets an operation mode according to the designation of a channel of an audio signal on which sound image localization signal processing is performed, and the viewpoint selection information of the user selected by the viewpoint selection unit 5. A track setting unit 32 that sets a corresponding frame image, and a parameter setting unit 33 that sets a parameter for performing signal processing on an audio signal corresponding to the selected frame image are configured.

The operation mode setting section 31 includes a fixed channel mode 31-1 as a first operation mode for performing sound image localization signal processing on all channels of audio signals of a plurality of channels and an audio signal of a plurality of channels. A non-fixed channel mode 31-2, which is a second operation mode in which only specific channels are designated and sound image localization signal processing is performed and other channels are not subjected to signal processing, is configured.

The track setting unit 32 includes a frame processing unit 32-1 for performing frame processing of the video signal so as to correspond to the user's viewpoint selection information which is the angle information of the reproduced sound image selected by the viewpoint selection unit 5. A horizontal angle processing unit 32-2 that performs horizontal angle processing, a zoom processing unit 32-3 that performs zoom processing, and a vertical angle processing unit 32 that performs vertical angle processing.
-3 and. The track setting unit 32 generates the operation signal C for performing signal processing on the audio signal according to the operation mode AM from the operation mode setting unit 31 and the viewpoint selection information V from the viewpoint selecting unit 5 with the above-described configuration. Note that all of these configurations may be used, or the present invention is not limited to this, and any one of them may be selected and used according to the processing.

The parameter setting section 33 performs an equalizing processing section 33-1 for performing equalizing processing of phase equalization on the audio signal with a predetermined parameter, and a reverberation reverb processing based on the operation signal C from the track setting section 32. The reverb processing unit 33-2 for performing, the volume processing unit 33-3 for performing the volume processing, the pan processing unit 33-4 for performing the pan processing of the movement of the sound image, and the HRTF (head related transfer function: Hea).
and a HRTF processing unit 33-5 that performs acoustic characteristic processing by changing the transfer function reaching the user's ear by using the dRelated Transfer Form.

All of these configurations may be used, or the present invention is not limited to this, and any one may be selected and used according to the processing. Here, in accordance with the operation signal C supplied to the parameter setting unit 33, the metadata MD that is the control information for each processing unit of the parameter setting unit 33 is generated. The generation of the metadata MD will be described later.

Here, the HRTF processing section will be described. For example, five sets of stereo sound source data can be obtained by using five stereo impulse responses from five sound sources placed around the listener to both ears of the listener. For example, the first sound source that convolves the transfer function of the impulse response from the sound source in the front left of the listener to both ears of the listener with the input signal, and the transfer of the impulse response from the sound source in the rear left of the listener to both ears of the listener A case where signal processing is performed on a second sound source that convolves a function with an input signal will be described.

When the creator inputs movement information for moving the sound image position from the first sound source to the second sound source by the viewpoint selection unit 5, the track setting unit 32 of the channel mapping unit 6 uses this movement information as an angle parameter or It is converted into a position parameter, and the operation signal C according to the converted parameter is supplied to the parameter setting unit 33.

With this operation signal C, the parameter setting unit 33 processes the level values of the respective processing units so as to crossfade with the ratio of the distance or the angle from the first sound source and the second sound source.

Next, the detailed structure and operation of the HRTF processor will be described. The left digital signal L of the digital signal string of the audio signal output from the mixer 3 is supplied to the pair of left convolutional integrators and the convolutional integrator of the memory. Here, in the memory attached to the convolutional integrator, the constant sampling frequency and the number of quantization bits from the virtual sound source position to both ears with respect to the reference direction of the head of the listener's current head are stored. The set of digitally recorded impulse responses represented is being recalled. The digital signal sequence is convolutionally integrated in real time with the impulse response read from this memory in the convolutional integrator. A pair of right crosstalk convolutional integrators and a memory supply the crosstalk component of the right digital signal R.

Similarly to the above, the right digital signal R is supplied to the convolutional integrator of the pair of right convolutional integrators and the memory. Here, in the memory attached to the convolutional integrator, the constant sampling frequency and the number of quantization bits from the virtual sound source position to both ears with respect to the reference direction of the head of the listener's current head are stored. A set of digitally recorded impulse responses represented is stored. The digital signal sequence is convolutionally integrated in real time with the impulse response read from this memory in the convolutional integrator. A pair of left crosstalk convolutional integrators and a memory supply the crosstalk component of the left digital signal L.

Also, in the pair of right crosstalk convolutional integrators and memories and the pair of left crosstalk convolutional integrators and memories, impulse response and convolutional integration are performed in the same manner as described above. In this way, the pair of left, the pair of right crosstalk, the pair of right, and the pair of left crosstalk convolutional integrators and the digital signal sequence on which the convolution integration is performed in the memory are respectively added to the adder. Supplied. The two-channel digital signals added by the adder are corrected by an adaptive processing filter so as to eliminate differences in ear shape due to individual differences of listeners, noise, characteristics peculiar to the sound source used, and the like.

In the above example, the example in which the impulse response as the HRTF is stored in the memory is shown, but from the virtual sound source position of the head fixed with respect to the reference direction, in the memory attached to the convolutional integrator. Store a pair of digitally stored impulse responses to both ears.
The digital signal sequence is convolved with this impulse response in real time. The other memory stores a control signal indicating a time difference and a level difference between both ears from the virtual sound source position with respect to the reference direction of the head to both ears.

Then, with respect to each of the digital signals of the respective channels subjected to the convolutional integration, the detected head movement with respect to the reference direction is further magnified to include a direction at a constant unit angle or at a predetermined angle. To a digital address signal indicating the level, the control signal stored in another memory in advance is read by this address signal, the control device corrects and changes in real time, and the result is supplied to the adder. It may be done.

The impulse response and the digital signal sequence convolutively integrated in real time are supplied to the adder, and the detected head movement in the reference direction is further detected for the 2-channel digital signal from the adder. Every fixed unit angle or every predetermined angle,
Alternatively, the control signal may be converted into a digital address signal representing a magnitude including a direction, the control signal stored in advance in another memory may be read by this address signal, and the control device may correct and change in real time.

Here, the control device can be constituted by a combination of a variable delay device and a variable level controller, or a level controller for each frequency band such as a graphic equalizer divided into multiple bands. Further, the information stored in the other memory represents a time difference and a level difference between the two ears from the virtual sound source position to both ears with respect to the reference direction of the head of the listener's head. Impulse response is also acceptable. In this case, the above-mentioned control device may be composed of an IIR or FIR variable digital filter. Therefore, using the controller, the HR
The value of impulse response as TF may be changed.

In this way, the controller gives spatial information, the adaptive processing filter corrects the difference in ear shape due to the individual difference of the listener, noise, the sound source to be used, and the peculiar characteristics of the headphones, and Change is given to movement.

In this way, the viewpoint selecting section 5 detects the head movement of the listener with respect to the reference direction at a constant angle or at each predetermined angle, and converts it into a digital address signal representing a magnitude including the direction in the address control circuit. To do.

With this address signal, the digitally recorded impulse response from the virtual sound source position with respect to the reference direction of the head recorded in the memory in advance to both ears, and the time difference between both ears and both ears are recorded from other memories. Read out a control signal or an impalas response indicating the level difference between the two. The convolutional integrator or control device corrects and changes the impalased response or control signal and the acoustic signal in real time.

The convolutional integrator and the memory or the control device and the adder convert the two-channel digital signals for both ears having the spatial information as the sound field, and the adaptive processing filter changes the ears depending on the individual difference of the listener. Differences in shape, noise, characteristics of the sound source used, etc. are corrected,
The power is amplified by the power amplifier and then supplied to the speaker. As a result, it is possible to realize a reproducing effect as if the reproducing sound is heard from the speaker placed at the virtual sound source position.

In this case, when the listener moves the free viewpoint video using the free viewpoint viewer 4, the viewpoint selecting unit 5
By this, a digital signal or an analog signal according to the direction is obtained, and the signal has a value according to the direction of the viewpoint of the listener with respect to the free viewpoint video. This value is supplied to the memory as an address signal through the address control circuit.

From the memory, among the data corresponding to the table, the digitally recorded impalares response from the virtual sound source position to both ears or the time difference between the ears with respect to the reference direction corresponding to the direction of the viewpoint of the listener with respect to the free viewpoint image. And a control signal representing the level difference between the two ears is taken out and this data is fed to a convolutional integrator or controller.

Based on the angle information from the viewpoint selection unit 5, a digitally recorded impalares response from the virtual sound source position to both ears corresponding to the direction of the listener's viewpoint with respect to the free viewpoint video from the memory to both ears or both. A control signal representing the time difference between the ears and the level difference between the ears is taken out, and this data is supplied to a convolution integrator or a controller.

In this way, the audio signals L and R supplied to the speaker are digitally recorded Impalares response from the virtual sound source position to both ears to the reference direction corresponding to the direction of the listener's viewpoint with respect to the free viewpoint video or Since the correction is performed with the control signals that represent the time difference between both ears and the level difference between both ears, even if the listener's viewpoint moves with respect to the free viewpoint video, a plurality of speakers generate virtual sound sources in the moving direction. It is possible to obtain a sound field feeling that the speaker is placed at a position and reproduced by this speaker.

Further, a control signal representing the time difference between both ears and the level difference between both ears, which is digitally recorded in the table of the memory, is taken out, and this data is converted into a digital signal pre-convoluted by the convolution integrator and the memory. for,
Since it is supplied purely electronically so as to be corrected by the control device, there is no delay in changing the characteristics of the audio signal with respect to the orientation of the listener's head, and no unnaturalness occurs.

Further, the HRTF data can be obtained as follows. That is, an impulse sound source having a required number of channels and a dummy head microphone are arranged at predetermined positions in an appropriate room so that a preferable reproduced sound field is obtained when an audio signal is reproduced by a speaker for a free viewpoint image. A speaker may be used as a sound source for measuring impulses in this case.

The sound collecting position of each ear of the dummy head may be any position from the entrance of the external auditory meatus to the eardrum position, but it is equal to the position for obtaining the correction characteristic for canceling the characteristic peculiar to the sound source used. Is required.

The control signal can be measured by radiating an impulse sound from the speaker position of each channel and collecting the sound at a fixed angle by a microphone provided in each ear of the dummy head. Therefore, at a certain angle, one set of impulse responses can be obtained for each channel. Therefore, in the case of a signal source of 5 channels, 5 sets, that is, 10 kinds of control signals are obtained for each angle. Will be done. Therefore, the control signals representing the time difference and the level difference between the left and right ears can be obtained from these responses.

Further, in the above description, only the orientation of the listener's head in the horizontal plane is considered, but the orientations in the vertical plane and in the plane orthogonal to these can also be processed in the same manner.

There is one table in the memory,
In the address control circuit, the designation of the address for the table can be changed to obtain control data as in the case where there are a plurality of sets of tables.

Further, the data in the table may be limited to the range of the direction of the viewpoint of the listener with respect to a general free viewpoint video, and the angle θ is, for example, 0.
The interval of the angle θ may be different depending on the direction, such as setting every 5 ° and setting every 3 ° when | θ ≧ 45 ° |. As described above, the angle may be such that the listener can identify the angle of rotation of the viewpoint of the listener with respect to the free viewpoint video. Further, instead of the headphones, a speaker placed near both ears of the listener may be used.

FIG. 4 shows a GUI (Graphical U).
It is a figure which shows a ser Interface application screen. FIG. 4 shows a display state of the free viewpoint viewer 4 by the viewpoint selecting unit 5 shown in FIG. 1 and a setting state of the channel mapping unit shown in FIG.
In FIG. 4, an operation mode setting unit 42 is provided on the GUI application screen 41, and when the sound creator clicks the icon of the operation mode setting unit 42, as shown in FIG. Fixed channel mode 31-1, which is the first operation mode in which sound image localization signal processing is performed on all channels of an audio signal, and sound image localization signal processing is performed by designating only a specific channel among audio signals of a plurality of channels. For other channels, the non-fixed channel mode 31-2 of the second operation mode in which the signal processing is not performed can be selected and set.

On the GUI application screen 41, a track setting section 43 is provided, and when the sound creator clicks the icon of the track setting section 43, the angle information of the reproduced sound image selected by the viewpoint selecting section 5 is displayed. A frame processing unit 43-1 that performs frame processing by moving a frame according to time information of a video signal so as to correspond to user's viewpoint selection information and a horizontal angle processing unit 43-2 that performs horizontal angle processing. The zoom processing unit 43-3 that performs zoom processing and the vertical angle processing unit 43-3 that performs vertical angle processing can be set to a predetermined frame image setting state. Note that, here, the track setting unit 43 corresponds to the viewpoint selecting unit 5 shown in FIG. 1, and the display state of the free viewpoint viewer 4 displayed in a window below or above this GUI application screen 41 according to this setting is It changes over 360 degrees of free-viewpoint video.

A parameter setting section 44 is provided on the GUI application screen 41, and when the sound creator clicks the icon of the parameter setting section 44, the track setting section 32 shown in FIG.
Based on the operation signal C from the equalizing processing unit 44-1 for performing equalizing processing on the audio signal with a predetermined parameter, a reverb processing unit 44-2 for performing reverb processing, and a volume processing unit 44-3 for performing volume processing.
The track setting unit 43 sets the pan processing unit 44-4 for performing the pan processing, and the HRTF processing unit 44-5 for performing the acoustic characteristic processing by changing the transfer function reaching the user's ear by the HRTF. It is possible to set a signal processing state for an audio signal for a frame image. Corrected audio data can be obtained according to the setting by the parameter setting unit 44.

The generation of metadata will be described in detail below. FIG. 5 is a diagram showing generation of operation metadata. The configuration of FIG. 5 is provided for each channel of the audio signal to be subjected to signal processing. In FIG. 5, the operation information C from the track setting unit 32 shown in FIG. 3 corresponds to the equalizer processing unit 33-1 and the reverb processing unit 3 of the parameter setting unit 33.
3-2, volume processing unit 33-3, pan processing unit 33-4, H
It is supplied to the RTF processing unit 33-5. Audio data AI is continuously input to each processing unit. Each processing unit performs signal processing based on the operation signal C and outputs corrected audio data AO.

At this time, for example, operation metadata for the equalizer processing unit 33-1 for lowering the equalizer processing in the high frequency band or the low frequency band according to the angle information of the operation information C, and for the reverb processing unit 33-2, for example, Operation metadata for reducing reverb processing in a high frequency band or a low frequency band according to the angle information of the operation information C, for example, movement of a sound position according to the angle information of the operation information C with respect to the pan processing unit 33-4. And the operation metadata for improving or reducing the frequency transfer characteristic according to the angle information of the operation information C for the HRTF processing unit 33-5, respectively, and operate in association with each angle information. It is stored in the metadata section MC.

As a result, it is possible to generate the operation metadata which can change the signal processing function for the audio signal in the channel mapping section according to the angle information of the visual field of the user.

The configuration of each of these processing units may be all, or not limited to this, and any one may be selected and used according to the process, but in this case, it is selected. The operation metadata of the processing unit is stored in the operation metadata unit MC in association with each angle information.

FIG. 6 is a diagram showing the generation of time metadata. The configuration shown in FIG. 6 is provided for each channel of an audio signal to be subjected to signal processing. 6, the operation information C from the track setting unit 32 shown in FIG. 3 corresponds to the equalizer processing unit 33-of the parameter setting unit 33, which is not shown here.
1, the reverb processing unit 33-2, and the volume processing unit 33-3, and then is supplied to the pan processing unit 33-4 and the HRTF processing unit 33-5 in FIG. Audio data AI is continuously input to each processing unit. Operation signal C in each processing unit
The corrected audio data A is subjected to signal processing based on
O is output.

In this case, time metadata for the equalizer processing unit 33-1 (not shown here) for reducing the equalizer processing in the high frequency band or the low frequency band corresponding to the time information of the operation information C, for example, the reverb processing unit 33. -2, for example, time metadata for reducing the reverb processing in the high frequency band or the low frequency band according to the time information of the operation information C, and the sound corresponding to the time information of the operation information C for the pan processing unit 33-4. Metadata for moving the position of HRTF, HRTF processing unit 33-5
For example, the time metadata for improving or decreasing the frequency transfer characteristic corresponding to the time information of the operation information C is extracted and stored in the time metadata portion MT in association with each time information.

As a result, it is possible to generate time metadata capable of changing the signal processing function for the audio signal in the channel mapping unit according to the time information due to the change of the time axis.

The configuration of each of these processing units may be all, or not limited to this, and any one may be selected and used according to the process, but in this case, it is selected. The time metadata of the processing unit is stored in the time metadata unit MT in association with each time information.

FIG. 7 is a diagram showing a recording format of metadata. FIG. 7A shows the beginning of one frame, and FIG.
It is recorded at the end of the frame, and at the beginning of each album in FIG. 7C. First, in the recording format of the first metadata, in FIG. 7A, the video data V1, V2, V
The metadata M1 is recorded at the beginning of one frame in which audio data is interleaved at 3, V4, and V5. Therefore, in the reproducing system, the metadata M1 can be read by detecting one frame of data, holding the data in the buffer, and reading the beginning portion thereof.

Next, in the recording format of the second metadata, in FIG. 7B, the video data V1, V2, V
The metadata M2 is recorded at the end of one frame in which audio data is interleaved at 3, V4, and V5. Therefore, in the reproducing system, the metadata M2 can be read by detecting one frame of data, holding it in the buffer, and reading the end portion thereof.

Further, in the third metadata recording format, in FIG. 7C, the metadata M11, M12 and M13 are recorded at the head of the album showing the music. Therefore, in the reproducing system, if the data of each album is detected and held in the buffer, and the beginning portion thereof is read, the metadata M
11, M12 and M13 can be read.

The recording format of the metadata is
The present invention is not limited to this, and may be recorded in a TOC (Table Of Contents) portion that stores management information of the disc-shaped recording medium.

FIG. 8 is a diagram showing an image picked up by a 360-degree camera. In FIG. 8, the omnidirectional camera 81 divides a 360-degree surrounding object into, for example, eight view angles, and continuously processes and captures the image. The 360 degree camera imaged image 82 imaged by the omnidirectional camera 81 includes an area E1 showing an image of the singer 84 and the performer 85 on the stage 83 having angle information θ in the front center direction with respect to the reference position O, and the reference. An area E2 showing an image of the spectator 86 having angle information θ in the front left direction with respect to the position O, an area E3 showing an image of the spectator 86 having angle information θ in the left lateral direction with respect to the reference position O, and the reference An area E4 showing an image of the spectator 86 having angle information θ in the left rear direction with respect to the position O, an area E5 showing an image of the spectator 86 having angle information θ in the rear center direction with respect to the reference position O, and an area E5. An area E6 showing an image of the spectator 86 having angle information θ in the right rear direction with respect to the position O, and a spectator 86 having angle information θ in the right lateral direction with respect to the reference position O.
And an area E7 showing an image of the spectator 86 having angle information θ in the front right direction with respect to the reference position O.

The production system channel mapping unit 6 shown in FIG. 1 subjects the audio signal to signal processing for sound image localization synchronized with the movement of the image in accordance with the angle information θ by the viewpoint selection unit 5 to obtain a 360-degree camera image. The sound image can be smoothly moved in the counterclockwise direction in response to the movement of the image in the counterclockwise direction according to the angle information θ from the area E1 to the area E8 of the image 82.

Thus, the singer 84 and the performer 84 on the stage 83 in the front center direction with respect to the images of the singer 84 and the performer 85 on the stage 83 having the angle information θ in the front center direction with respect to the reference position O in the area E1. 85
Sound image is localized, and the sound image of the spectator 86 in the front left direction is localized with respect to the image of the spectator 86 having angle information θ in the front left direction with respect to the reference position O in the area E2.
In the area E, the sound image of the spectator 86 in the left lateral direction is localized with respect to the image of the spectator 86 in the left lateral direction with respect to the reference position O.
Angle information θ in the rear left direction with respect to the reference position O
Audience 86 to the rear left of the image of the audience 86
Sound image is localized and the sound image of the spectator 86 in the rear center direction is localized with respect to the image of the spectator 86 having the angle information θ in the rear center direction with respect to the reference position O in the area E5, and the rear right with respect to the reference position O in the area E6. The spectator 8 in the rear right direction with respect to the image of the spectator 86 having the angle information θ of the direction
The sound image of 6 is localized, and the sound image of the spectator 86 in the right lateral direction is localized with respect to the image of the spectator 86 having the angle information θ in the right lateral direction with respect to the reference position O in the area E7. The sound image of the spectator 86 in the front right direction is continuously localized with respect to the image of the spectator 86 having the right angle information θ.

On the contrary, the sound image can be smoothly moved in the clockwise direction corresponding to the movement of the image in the clockwise direction according to the angle information θ from the area E1 to the area E2 of the 360 ° camera image 82.

As a result, the singer 84 and the performer on the stage 83 in the front center direction with respect to the images of the singer 84 and the performer 85 on the stage 83 having the angle information θ in the front center direction with respect to the reference position O in the area E1. 85
Sound image is localized, and the sound image of the spectator 86 in the front right direction is localized with respect to the image of the spectator 86 having angle information θ in the front right direction with respect to the reference position O in the area E8.
In the image of the spectator 86 having the angle information θ in the right lateral direction with respect to the reference position O, the sound image of the spectator 86 in the right lateral direction is localized, and the spectator having angle information θ in the right rear direction with respect to the reference position O in the area E6 The sound image of the spectator 86 in the rear right direction is localized with respect to the image of 86, and the sound image of the spectator 86 in the rear center direction is detected with respect to the image of the spectator 86 having the angle information θ in the rear center direction with respect to the reference position O in the area E5. The sound image of the spectator 86 in the rear left direction is localized with respect to the image of the spectator 86 that is localized and has angle information θ in the rear left direction with respect to the reference position O in the area E4, and the spectator in the left lateral direction with respect to the reference position O in the area E3 A sound image of the spectator 86 in the left lateral direction is localized with respect to the image of 86, and an image of the spectator 86 having angle information θ in the front left direction with respect to the reference position O in the area E2 is obtained. On the other hand, the sound image of the spectator 86 in the front left direction is continuously localized.

Further, the angle information at this time is not limited to the angle information θ in the horizontal direction, but the angle information Δ in the vertical direction can be designated. As a result, the image captured by the 360-degree camera 8
The localization of the sound image can be processed in the three-dimensional space by the channel mapping unit for the images of the areas E1 to E8 of No. 2.

The control amount of each signal processing is as follows.
According to the angle information θ that moves with respect to the reference position O of 1.
The pan processing unit 33-4 illustrated in FIG. 3 performs processing so that the localization angle of the sound image moves proportionally.

When the omnidirectional camera 81 moves from the front to the rear or from the back to the front with respect to the reference position O, the volume of the sound image is reduced or increased by the volume processing unit 33-3 shown in FIG. 3 according to the moving distance. Will be processed.

When the omnidirectional camera 81 moves from the front to the rear or from the rear to the front with respect to the reference position O, the equalizer processing unit 33-1 shown in FIG. It is processed to lower the sound in the high frequency band or to improve the equalizer processing in the high frequency band to raise the sound in the high frequency band.

When the omnidirectional camera 81 moves from the front to the rear or from the rear to the front with respect to the reference position O, the reverb processing unit 33-2 shown in FIG. 3 applies deep reverb processing according to the moving distance. Alternatively, the reverb process is processed so that it takes a shallow depth.

When the omnidirectional camera 81 moves from the front to the rear or the back to the front by the zoom process with respect to the reference position O, the HRTF processor 33-5 shown in FIG. The value is changed so that the value is changed so that the sound image becomes closer.

Incidentally, not only the omnidirectional camera but also a circumferential camera described later may be used. FIG. 9 is a diagram showing imaging by a circumferential camera. In FIG. 9, for the singer 92, which is the subject, according to the angle information θ with respect to the reference position O, the circumferential cameras 91-1, 91-2, 91-3, 91.
-4, 91-5, 91-6, 91-7, 91-8 are provided to capture the circumferential image data having the angle information θ.
In addition, not only the circumferential camera but also a part of the arc camera may be used. Also in these cases, the angle information at this time is not limited to the angle information θ in the horizontal direction, and the angle information Δ in the vertical direction can be designated. Accordingly, the localization of the sound image can be processed in the three-dimensional space by the channel mapping unit with respect to the image of each area of the circumferentially or arcuate imaged image.

FIG. 10 is a diagram showing switching of operation modes on the time axis. 10, mode 1 (101), mode 2 (102), mode 1 (103), ... Are switched along the time axis t. Here, mode 1 (101) and mode 1 (103) are sound image localization signals for all channels set by the operation mode setting unit 31 of the channel mapping unit shown in FIG. This is a fixed channel mode 31-1 which is a first operation mode for performing processing, and a mode 2 (102) specifies only a specific channel among audio signals of a plurality of channels, performs sound image localization signal processing, and performs other processing. The channel is the non-fixed channel mode 31-2 which is the second operation mode in which no signal processing is performed.

As a result, the fixed channel mode 31-1 of the first operation mode and the non-fixed channel mode 31-2 of the second operation mode can be switched on the time axis, and the scene of the free viewpoint video can be switched. It is possible to diversify the mode of sound image localization processing for an audio signal in time. For example, as the fixed channel mode 31-1 of the first operation mode, the singer 84 and the performer 8 in the 360-degree camera imaged image 82 shown in FIG.
The sound image localization signal processing is performed on all channels of the 5 audio signals, and as the non-fixed channel mode 31-2 of the second operation mode, the audio of the singer 84 in the 360-degree camera imaged image 82 shown in FIG. Performing sound image localization signal processing only on the signal channel, the performer 85
The audio signal channel is fixed without being subjected to sound image localization signal processing.

FIG. 11 is a diagram showing a corrected sound image by HRTF. In FIG. 11, the listener 111 during reproduction is
The reproduced sound image 113 reproduced by the speakers L, R, SL, SR is heard corresponding to the angle information of the free viewpoint video reproduced on the video monitor 112. Here, when the free-viewpoint image reproduced on the video monitor 112 becomes a zoom image, for example, when the singer on the stage approaches the right ear of the listener 111 and whispers, the HRTF processing unit 33-5 shown in FIG.
Is processed so that the sound image is closer to the right ear of the listener 111.
The corrected sound image 114 is processed so as to be localized at the right ear.

FIG. 12 is a block diagram of correction processing by HRTF. In FIG. 12, the audio data AI is supplied to the pan processing unit 121, is panned by the pan processing unit 121, and is left / right stereo audio data R, L.
And surround audio data SL and SR are output. In this state, the switches SW123 and SW124
The output side of is connected to the pan processing unit 121 side.

Here, when the free-viewpoint image reproduced on the video monitor 112 becomes a zoom image, for example, when the singer on the stage approaches the right ear of the listener 111 and whispers, the switch SW11 in which the audio data AI is turned on is turned on. Is supplied to the HRTF processing unit 122 via the HRTF processing unit 122, subjected to frequency transfer characteristic processing by the HRTF processing unit 122, supplied to the switches SW123 and 124, and the left and right stereo audio data R and L are output as the corrected audio data AO. In this state, the switches SW123 and SW1
The output side of 24 is connected to the HRTF processing section side.

The operations of the production system and the reproduction system of the free-viewpoint video / audio production / reproduction system thus configured will be described. FIG. 13 is a flowchart showing the operation of the production system. In FIG. 13, in step S1, the input / output device (I
O device) is initialized. Specifically, the assignment of the initial value or, for example, GU on the operating system (OS)
The I application is started to read the control program.

In step S2, a video file is designated. Specifically, a video file for producing content is designated from the video data 10 of the free viewpoint video shown in FIG.

In step S3, the operation mode is designated. Specifically, when the sound creator clicks the icon of the operation mode setting unit 42 on the GUI application screen 41 shown in FIG. 4, all the channels of the audio signals of a plurality of channels are displayed as shown in FIG. The fixed channel mode 31-1 of the first operation mode in which the sound image localization signal processing is performed or only a specific channel of the audio signals of a plurality of channels is designated to perform the sound image localization signal processing and the other channels are subjected to the signal processing. The non-fixed channel mode 31-2, which is the second non-fixed operation mode, is selected and set.

In step S4, a fixed channel or a mobile channel is assigned. Specifically, first, G
When the sound creator clicks the icon of the track setting unit 43 on the UI application screen 41, the time of the video signal is adjusted so as to correspond to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected by the viewpoint selection unit 5. A frame processing unit 43-1 that performs frame processing by moving a frame according to information, a horizontal angle processing unit 43-2 that performs horizontal angle processing, a zoom processing unit 43-3 that performs zoom processing, and a vertical angle processing. And the vertical angle processing unit 43-3 for performing the setting are set to the setting state of the selected frame image. Note that, here, the track setting unit 43 corresponds to the viewpoint selecting unit 5 shown in FIG. 1, and the free viewpoint video of the free viewpoint viewer 4 displayed in the window below or above the GUI application screen 41 according to this setting. Is set to the setting state of the selected frame image according to the display state changing over 360 degrees.

Next, when the sound creator clicks the icon of the parameter setting section 44 on the GUI application screen 41, a predetermined parameter is added to the audio signal based on the operation signal C from the track setting section 32 shown in FIG. The equalizing processing section 44-1 for performing the equalizing processing, the reverb processing section 44-2 for performing the reverb processing, and the volume processing section 44 for performing the volume processing.
-3, a pan processing unit 44-4 that performs pan processing, and an HRT
An HRTF processing unit 44-5, which performs acoustic characteristic processing by changing the transfer function reaching the user's ear by F, and a signal processing state for the audio signal of each channel for the frame image set by the track setting unit 43. Can be set to. The mobile channel or the fixed channel is set according to the setting by the parameter setting unit 44.

At step S5, the reproduction of the image is started. Specifically, the reproduction of the video by the video signal which changes the free viewpoint video of the free viewpoint viewer 4 displayed in the lower layer or the upper layer of the GUI application screen 41 over 360 degrees to be in the display state is started. At the same time, reproduction of sound by the audio signal is started.

In step S6, it is determined whether or not the time metadata recording mode when the time key 45, which is the time metadata recording mode key, is pressed. Specifically, the parameter setting unit operates metadata for lowering the equalizer process in the high frequency band or the low frequency band corresponding to the angle information of the operation information C for the equalizer processing unit 33-1 illustrated in FIG. 5, for example. , Reverb processing unit 3
3-2, for example, operation metadata for reducing reverb processing in a high frequency band or a low frequency band according to angle information of the operation information C, and for example, according to angle information of the operation information C for the pan processing unit 33-4. There is operation metadata for moving the position of sound, signal processing by operation metadata for improving or reducing frequency transfer characteristics according to the angle information of the operation information C for the HRTF processing unit 33-5, or The parameter setting unit, for the equalizer processing unit 33-1 illustrated in FIG. 6, for example, time metadata for lowering the equalizer processing in the high frequency band or the low frequency band according to the time information of the operation information C, the reverb processing unit. 33-2, for example, operation information C
Time metadata for lowering the reverb processing in the high frequency band or the low frequency band according to the time information, for moving the position of the sound according to the time information of the operation information C with respect to the pan processing unit 33-4, for example. It is determined whether or not it is due to the time metadata or the time metadata for improving or decreasing the frequency transfer characteristic according to the time information of the operation information C for the HRTF processing unit 33-5.

If the mode is not the time metadata recording mode in step S6, the process proceeds to step S7, and in step S,
It is determined whether or not there is an operation of the channel mapping signal processing unit based on the operation metadata. Specifically, operation metadata for reducing the equalizer processing in the high frequency band or the low frequency band corresponding to the angle information of the operation information C for the equalizer processing unit 33-1 illustrated in FIG. 5, the reverb processing unit 33, for example. -2, for example, operation information C
Operation metadata for lowering the reverb processing in the high frequency band or the low frequency band according to the angle information, for moving the position of the sound according to the angle information of the operation information C with respect to the pan processing unit 33-4, for example. Operation metadata, HR
For example, it is determined whether the TF processing unit 33-5 has signal processing by operation metadata for improving or reducing the frequency transfer characteristic according to the angle information of the operation information C.

If there is an operation of the channel mapping signal processing section by the operation metadata in step S7, the process proceeds to step S8, and the viewpoint information and the operation information are recorded as metadata in step S8. Specifically, operation metadata for reducing the equalizer processing in the high frequency band or the low frequency band corresponding to the angle information of the operation information C for the equalizer processing unit 33-1 illustrated in FIG. 5, the reverb processing unit 33, for example. -2, for example, operation information C
Operation metadata for lowering the reverb processing in the high frequency band or the low frequency band according to the angle information, for moving the position of the sound according to the angle information of the operation information C with respect to the pan processing unit 33-4, for example. Operation metadata, HR
For example, the operation metadata for improving or reducing the frequency transfer characteristic corresponding to the angle information of the operation information C for the TF processing unit 33-5 is extracted and stored in the operation metadata unit MC in association with each angle information. .

If the time metadata recording mode is selected in step S6, the process proceeds to step S10 and step S1.
At 0, it is determined whether or not there is an operation of the channel mapping signal processing unit by the time metadata. Specifically, for example, time metadata for reducing the equalizer processing in the high frequency band or the low frequency band corresponding to the time information of the operation information C for the equalizer processing unit 33-1 illustrated in FIG. 6, and the reverb processing unit 33. -2, for example, time metadata for reducing the reverb processing in the high frequency band or the low frequency band according to the time information of the operation information C, and the sound corresponding to the time information of the operation information C for the pan processing unit 33-4. Metadata for moving the position of, for example, operation information C for the HRTF processing unit 33-5.
The time metadata for improving or lowering the frequency transfer characteristic according to the time information is extracted, and it is determined whether or not the time metadata is stored in the time metadata portion MT in association with each time information.

At step S10, if there is an operation of the channel mapping signal processing section by time metadata,
In step S11, time information and operation information are recorded as metadata. Specifically, for example, time metadata for reducing the equalizer processing in the high frequency band or the low frequency band corresponding to the time information of the operation information C for the equalizer processing unit 33-1 illustrated in FIG. 6, and the reverb processing unit 33. -2, for example, time metadata for lowering the reverb processing in the high frequency band or the low frequency band according to the time information of the operation information C, the pan processing unit 33-
4, time metadata for moving the position of the sound according to the time information of the operation information C, the HRTF processing unit 33.
For example, the time metadata for improving or decreasing the frequency transfer characteristic corresponding to the time information of the operation information C for -5 is extracted and stored in the time metadata portion MT in association with each time information.

In step S9, it is determined whether or not the image reproduction is stopped. If the video reproduction is not stopped in step S9, the process returns to step S6 and step S6 is performed.
Repeat judgment 6 above. When the reproduction of the video is stopped in step S9, the process proceeds to step S12, and the reproduction of the video and the sound is stopped in step S12.

FIG. 14 is a flow chart showing the operation of the reproducing system. In FIG. 14, in step S21, the input / output device (IO device) is initialized. In particular,
The control program is read by substituting an initial value or activating, for example, a GUI application on the operating system (OS).

In step S22, a video file is designated. Specifically, a video file for reproducing the content is designated from the video data of the video player 22 shown in FIG.

At step S23, the reproduction of the image is started. Specifically, the reproduction operation of the video player 22 starts the reproduction of the image by the image signal corresponding to the angle information selected by the viewpoint selection unit 23 among the free viewpoint images displayed on the video monitor 26. At the same time, reproduction of sound by the audio signal is started.

In step S24, it is determined whether or not the reproduction mode is the time metadata reproduction mode. Specifically, the parameter setting unit operates metadata for lowering the equalizer process in the high frequency band or the low frequency band corresponding to the angle information of the operation information C for the equalizer processing unit 33-1 illustrated in FIG. 5, for example. , Operation metadata for lowering the reverb processing in the high frequency band or the low frequency band according to the angle information of the operation information C, for example, to the reverb processing unit 33-2, and the angle of the operation information C for the pan processing unit 33-4, for example. The operation metadata for moving the position of the sound according to the information, the signal processing by the operation metadata for improving or reducing the frequency transfer characteristic according to the angle information of the operation information C for the HRTF processing unit 33-5. If the parameter setting unit is the operation information C for the equalizer processing unit 33-1 shown in FIG. Time metadata for lowering the equalizer processing in the high frequency band or the low frequency band according to the information, reverb processing in the high frequency band or the low frequency band according to the time information of the operation information C for the reverb processing unit 33-2, for example Metadata for lowering the pan, pan processing unit 33
-4, for example, time metadata for moving the position of the sound according to the time information of the operation information C, or improvement or deterioration of the frequency transfer characteristic according to the time information of the operation information C for the HRTF processing unit 33-5. To determine if it is due to time metadata.

In step S24, when it is not in the reproduction mode by the time metadata, the process proceeds to step S25, and in step S25, if it matches the viewpoint information, the operation information is sent to the signal processing unit to be changed. Specifically, in the reproduction mode based on the operation metadata, according to the angle information of the operation information C, for example, with respect to the equalizer processing unit 33-1 illustrated in FIG. 5, so as to correspond to the angle information selected by the viewpoint selecting unit 23. Operation metadata for reducing equalizer processing in a high frequency band or a low frequency band, reverb processing unit 3
3-2, for example, operation metadata for reducing reverb processing in a high frequency band or a low frequency band according to angle information of the operation information C, and for example, according to angle information of the operation information C for the pan processing unit 33-4. Signal processing is performed using operation metadata for moving a sound position, and operation metadata for improving or reducing frequency transfer characteristics according to, for example, angle information of the operation information C for the HRTF processing unit 33-5.

In step S24, if the reproduction mode is based on the time metadata, the process proceeds to step S27, and in step S27, the operation information is sent to the signal processing unit to be changed at the time. Specifically, in the reproduction mode based on time metadata, for example, a high frequency band or a low frequency band corresponding to the time information of the operation information C for the equalizer processing unit 33-1 illustrated in FIG. Time metadata for lowering the equalizer processing in the band, for example, operation information C for the reverb processing unit 33-2
Time metadata for lowering the reverb processing in the high frequency band or the low frequency band according to the time information, for moving the position of the sound according to the time information of the operation information C with respect to the pan processing unit 33-4, for example. Time metadata, HR
For example, signal processing is performed on the TF processing unit 33-5 with time metadata for improving or decreasing the frequency transfer characteristic according to the time information of the operation information C.

In step S26, it is determined whether or not the reproduction of the video is stopped. When the reproduction of the video is not stopped in step S26, the process returns to step S24 and the determination in step S24 is repeated. When the reproduction of the video is stopped in step S26, the process proceeds to step S28, and the reproduction of the video and the sound is stopped in step S282.

FIG. 15 is a diagram showing the function of channel mapping in the production system. FIG. 15A shows a case of real sound, FIG. 15B shows a case of virtual sound, and FIG. 15C shows a case of combination of real sound and virtual sound. .

In FIG. 15A, in the case of real sound, the operation information C is used to decrease the equalizer processing in the high frequency band or the low frequency band for the equalizer processing section 33-1 and the high frequency band for the reverb processing section 33-2 or Signal processing is performed by lowering the reverb processing in the low frequency band, lowering the volume for the volume processing unit 33-3, and moving the position of the surround sound with respect to the surround pan processing unit 151. Thereby, for example, real surround multi-output of 4 channels or 5 channels can be obtained.

In FIG. 15B, in the case of virtual sound, the equalizer processing unit 33-
1, a decrease in equalizer processing in a high frequency band or a low frequency band, a decrease in reverb processing in a high frequency band or a low frequency band in a reverb processing unit 33-2, a decrease in sound volume in a sound volume processing unit 33-3, and HR
Signal processing is performed by improving or decreasing the frequency transfer characteristic with respect to the TF processing unit 33-5. Thereby, for example, the reproduced sound image can be localized at an arbitrary position with respect to the channel of the selected and set audio signal.

In FIG. 15C, in the case of a combination of real sound and virtual sound, the switch SW
According to the switching of 21, SW22, SW23, and SW24, the operation information C is used to decrease the equalizer processing in the high frequency band or the low frequency band for the equalizer processing unit 33-1 and the high frequency band or the low frequency for the reverb processing unit 33-2. A case where a real surround output is obtained by performing signal processing by lowering the reverb processing in the frequency band, lowering the volume for the volume processing unit 33-3, and moving the position of the surround sound with respect to the surround pan processing unit 151, and the operation information. C reduces the equalizer processing in the high frequency band or the low frequency band with respect to the equalizer processing unit 33-1 and the reverb processing unit 33-
The signal processing is performed by reducing the reverb processing in the high frequency band or the low frequency band for 2, reducing the volume of the volume processing unit 33-3, and improving or reducing the frequency transfer characteristic of the HRTF processing unit 33-5.

With this, it is possible to selectively switch between the case of obtaining the output of the virtual surround,
While playing the real surround, SW21,
By switching SW22, SW23, and SW24, the reproduced sound image can be localized at an arbitrary position only for the channel of the audio signal that is selected and set.

Also at this time, for example, based on the operation information C, the viewpoint operation metadata 155 for reducing the sound by 3 dB when the angle information is 45 degrees is recorded, and based on the operation information C and the time information 154. Time metadata for switching from mode 1 to mode 2 may be recorded in 1 minute 30 seconds 15 frames.

In the case of FIGS. 15A and 15B as well, the operation metadata or the time metadata may be recorded.

16A and 16B are diagrams showing variations of channel mapping between the production system and the reproduction system. FIG. 16A shows a case of impulse response convolution type, and FIG. 16B shows a case of reverb processing compatible with 5ch output. In FIG. 16A, in the case of the impulse response convolution type, the equalizer processing is reduced in the high frequency band or the low frequency band with respect to the equalizer processing section 33-1 and the volume processing section 33-3.
To the impulse response convolution-type surround pan processing unit 161. Since the reverb processing unit is included, it is not necessary to provide the reverb processing unit. This makes it possible to obtain a 5-channel output having a corrected surround channel.

In FIG. 16B, in the case of reverb processing for 5ch output, the equalizer processing in the high frequency band or the low frequency band for the equalizer processing section 33-1 is reduced, and the volume processing section 33-3 is reduced in volume.
The reverb processing unit 33 is parallelly used by using the adders 163-167 with respect to the movement of the position of the surround sound in which the impulse response to the surround pan processing unit 161 is convoluted.
It is possible to obtain a 5-channel output having a corrected surround channel by performing reverb processing on each channel by performing signal processing by adding due to the reduction of the reverb processing in the high frequency band or the low frequency band with respect to −2.

Needless to say, the present invention is not limited to the examples shown in the above-mentioned embodiments, and other examples can be appropriately used within the scope of the claims of the present invention.

[0124]

According to the sound image localization signal processing apparatus of the present invention, the production system is arranged so that the reproduced sound image in the reproduction system is localized at an arbitrary position so as to correspond to the image based on the image signal having the angle information with respect to the reference position. In a sound image localization signal processing device that performs signal processing on an audio signal obtained by synthesizing sound source data input from a plurality of sound sources by a synthesizing means, user's viewpoint selection information serving as angle information of a reproduced sound image of a video based on the video signal. And a sound image localization control process for performing in real time signal processing for sound image localization on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected by the selection device. Means, and control information generation means for generating control information for performing signal processing by the sound image localization control processing means A transmission information generating means for generating transmission information by superimposing the audio signal and the control information on the video signal, and performing signal processing on the audio signal in real time to correspond to the video based on the video signal. Since I tried to control the sound image localization position,
When playing back audio signals for free-viewpoint video,
Even if the image is moved by changing the angle and reproduced, the control for localizing the sound image in the direction of the moving image can be performed at the time of production.

Further, in the sound image localization signal processing device of the invention, in the above description, the sound image localization control processing means performs signal processing by the sound image localization control processing means on all channels of the audio signals of a plurality of channels. The second operation mode, and the second operation in which only specific channels of the audio signals of a plurality of channels are designated and signal processing is performed by the sound image localization control processing means and the other channels are not subjected to the signal processing. Since there is an operation mode setting means for setting the mode, there is an effect that it is possible to perform sound image localization for each channel according to the operation mode.

Further, in the sound image localization signal processing device of the present invention, in the above description, the sound image localization control processing means corresponds to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected by the selection means. Selected by the selecting means because it has a track setting means for generating an operation signal for performing the signal processing on the audio signal by frame processing, horizontal angle processing, zoom processing and / or vertical angle processing of the video signal. The present invention has an effect that it is possible to perform signal processing on an audio signal by an operation signal generated by processing an image of a video signal so as to correspond to user's viewpoint selection information which is angle information of a reproduced sound image. .

Further, in the sound image localization signal processing device of the present invention, in the above description, the sound image localization control processing means, based on the operation signal from the track setting means, equalizes the audio signal with a predetermined parameter, Since it has the parameter setting means for performing the reverb processing, the volume processing, the pan processing and / or the transfer characteristic processing, it is possible to change the sound image localization by performing the signal processing on the audio signal according to the parameter.

Further, in the sound image localization signal processing device of the present invention, in the above, the control information generating means generates the control information by the parameter corresponding to the viewpoint selection information, so that the control information corresponding to the viewpoint selection information is generated. This has the effect of changing the sound image localization.

Further, in the sound image localization signal processing device of the present invention, in the above description, the control information generating means localizes the sound image by the control information corresponding to the time information for generating the control information by the parameter corresponding to the time axis information. The effect that it can be changed is produced.

Further, in the sound image localization signal processing device of the present invention, in the above description, the sound image localization control processing means sets the first operation mode and the second operation mode on the time axis by the operation mode setting means. Since it is switched with, the fixed channel mode of the first operation mode and the second
The operation mode can be switched to the non-fixed channel mode, and the mode of the sound image localization processing for the audio signal at the time of switching the scene of the free viewpoint video can be diversified.

Further, in the sound image localization signal processing device of the present invention, in the above description, the sound image localization control processing means performs the signal processing by designating only the virtual surround channel among the audio signals of a plurality of channels. Since other real surround channels do not perform the above signal processing, it is possible to selectively switch between the case of obtaining the output of virtual surround, so by switching during real surround playback, The reproduced sound image can be localized at an arbitrary position only for the channel of the audio signal that is selected and set.

Further, the sound image localization signal processing device of the present invention synthesizes each sound source data input from a plurality of sound sources by the synthesizing means and performs signal processing by the control information in the production system, at the reference position. On the other hand, in the sound image localization signal processing device for performing signal processing on the reproduction information so as to localize the reproduction sound image in the reproduction system so as to correspond to the image based on the video signal having the angle information, in the reproduction information, Reproduction information reading means for reading the video signal, the audio signal and the control information from the audio signal and the control information superimposed on the video signal, a video signal reproducing means for reproducing the video signal, and the video signal Selecting means for selecting the viewpoint selection information of the user, which is the angle information of the reproduced sound image of the video based on Audio signal localization control processing means for performing in real time signal processing for sound image localization on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected by the selection means. Since the sound image localization position is controlled according to the video based on the above video signal by performing signal processing in real time, in the playback of the audio signal for the free-viewpoint video, the video is moved by changing the angle and played back. Even in this case, there is an effect that the sound image can be localized in the direction of the moving image.

Further, in the sound image localization signal processing device of the present invention, in the above description, the sound image localization control processing means is a parameter for performing at least surround pan processing on the audio signal by a predetermined parameter, corresponding to the viewpoint selection information. Since the setting means is provided, it is possible to obtain an output having a corrected surround channel.

Further, according to the sound image localization signal processing method of the present invention, the production system is arranged so that the reproduced sound image in the reproduction system is localized at an arbitrary position so as to correspond to the image based on the image signal having the angle information with respect to the reference position. In the sound image localization signal processing method for performing signal processing on an audio signal obtained by synthesizing sound source data input from a plurality of sound sources by a synthesizing means, user's viewpoint selection information serving as angle information of a reproduced sound image of a video based on the video signal. And a sound image localization control process for performing, in real time, signal processing for sound image localization on the channel of the audio signal corresponding to the user's viewpoint selection information that is the angle information of the reproduced sound image selected by the selection step. And control information for performing signal processing by the sound image localization control processing step. A control information generation step; and a transmission information generation step of superimposing the audio signal and the control information on the video signal to generate transmission information, wherein the video signal is subjected to signal processing in real time. Since the sound image localization position is controlled in accordance with the video based on, the sound image is moved in the direction of the moving image even when the image is moved by changing the angle when reproducing the audio signal for the free viewpoint image. This has the effect that control for localizing can be performed during production.

Further, according to the sound image localization signal processing method of the present invention, the audio signals obtained by synthesizing the sound source data input from a plurality of sound sources by the synthesizing means and performing the signal processing by the control information in the production system are set at the reference position. On the other hand, in the sound image localization signal processing method for performing signal processing on the reproduction information so as to localize the reproduction sound image in the reproduction system so as to correspond to the video based on the video signal having the angle information, in the reproduction information, A reproduction information reading step for reading the video signal, the audio signal and the control information from the audio signal and the control information superimposed on the video signal; a video signal reproducing step for reproducing the video signal; and the video signal Selection step to select the user's viewpoint selection information, which is the angle information of the reproduced sound image of the video based on And a sound image localization control processing step for performing in real time signal processing for sound image localization on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected in the selection step. Since the audio signal is subjected to signal processing in real time to control the sound image localization position corresponding to the image based on the image signal, the angle of the image can be changed during the reproduction of the audio signal with respect to the free viewpoint image. Even when the image is moved and reproduced, there is an effect that the sound image can be localized in the direction of the moving image.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a sound image localization signal processing device of a production system of a free-viewpoint video / audio production / reproduction system applied to this embodiment.

FIG. 2 is a block diagram showing a configuration of a sound image localization signal processing device of a reproduction system of a free-viewpoint video / audio production / reproduction system applied to the present embodiment.

FIG. 3 is a block diagram showing a configuration of a channel mapping unit.

FIG. 4 is a diagram showing a GUI application screen.

FIG. 5 is a diagram showing generation of operation metadata.

FIG. 6 is a diagram showing generation of time metadata.

7A and 7B are views showing a recording format of metadata. FIG. 7A shows the beginning of one frame, FIG. 7B shows the end of one frame, and FIG. 7C shows the beginning of each album.

FIG. 8 is a diagram showing an image captured by a 360-degree camera.

FIG. 9 is a diagram showing imaging by a circumferential camera.

FIG. 10 is a diagram showing switching of operation modes on a time axis.

FIG. 11 is a diagram showing a corrected sound image by HRTF.

FIG. 12 is a configuration diagram of a correction process by HRTF.

FIG. 13 is a flowchart of a production system.

FIG. 14 is a flowchart of a reproduction system.

FIG. 15 is a diagram showing a function of channel mapping in a production system, and FIG. 15A shows a case of real sound, and FIG.
Shows a case of virtual sound, and FIG. 15C shows a case of combination of real sound and virtual sound.

16A and 16B are diagrams showing a variation of channel mapping of a production system and a reproduction system, FIG. 16A shows a case of an impulse response convolution type, and FIG. 16B shows a case of reverb processing compatible with 5ch output.

[Explanation of symbols]

1 ... Microphone material for free-view camera, 2 ... Other materials, 3 ... Mixer, 4 ... Free-view viewer, 5 ... View selection section, 6 ... Channel mapping section, 7 ... Audio monitor, 8 ...... Audio recorder, 9 ... Metadata, 10 ... Video data, 11 ... Audio interleave, 12 ... Sending data, 13 ... Network, 14 ... Recording medium, 21 ... Playing data, 22
...... Video player, 23 ...... Viewpoint selection section, 24 ...... Channel mapping section, 25 ...... Metadata, 26 ......
Video monitor, 27 ... Audio monitor, 31 ... Operation mode setting unit, 32 ... Track setting unit, 33 ... Parameter setting unit, 31-1 ... Fixed channel mode,
31-2 ... Non-fixed channel mode, 32-1, ... Frame processing unit, 32-2 ... Horizontal angle processing unit, 32-3
...... Zoom processing unit, 32-4 …… Vertical angle processing unit, 33
-1 ... equalizer processing unit, 33-2 ... reverb processing unit, 33-3 ... acoustic processing unit, 33-4 ... pan processing unit, 33-5 ... HRTF processing unit, 41 ... GUI application screen , 42 ... Operation mode setting section, 43 ...
… Track setting section, 44 …… Parameter setting section, C ……
Operation information, MC ... Operation metadata, MT ... Time metadata, 81 ... Omnidirectional camera, 82 ... 360-degree camera image, 91 ... Circular camera

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] March 31, 2003 (2003.3.3)
1)

[Procedure Amendment 1]

[Document name to be amended] Statement

[Name of item to be amended] Claims

[Correction method] Change

[Correction content]

[Claims]

10. An audio signal obtained by synthesizing sound source data input from a plurality of sound sources by a synthesizing means and performing signal processing by control information in a production system, based on a video signal having angle information with respect to a reference position. In a sound image localization signal processing device for performing signal processing on reproduction information so as to localize a reproduction sound image at an arbitrary position in a reproduction system so as to correspond to an image, the audio signal and the audio signal superimposed on the video signal in the reproduction information and Reproduction information reading means for reading the video signal, the audio signal and the control information from the control information, a video signal reproducing means for reproducing the video signal, and angle information of a reproduced sound image of the video based on the video signal. Selecting means for selecting the viewpoint selection information of the user, and the reproduced sound image selected by the selecting means. Sound signal localization control processing means for performing signal processing for sound image localization in real time on the channel of the audio signal corresponding to the user's viewpoint selection information that is the degree information, and performing signal processing on the audio signal in real time. A sound image localization signal processing device, characterized in that a sound image localization position is controlled corresponding to a video image based on the video signal.

In the sound image localization signal processing device 11. The method of claim 10, wherein said sound image localization control processing means, in correspondence with the view point selection information, the parameter setting for performing at least surround pan processed by predetermined parameters to the audio signal A sound image localization signal processing device comprising means.

12. A sound source input from a plurality of sound sources in a production system so that a reproduced sound image in a reproduction system is localized at an arbitrary position so as to correspond to a video based on a video signal having angle information with respect to a reference position. In a sound image localization signal processing method for performing signal processing on an audio signal obtained by synthesizing data by a synthesizing means, a selecting step for selecting viewpoint selection information of a user, which is angle information of a reproduced sound image of a video based on the video signal, and the selecting step. The sound image localization control processing step of performing signal processing for sound image localization in real time on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected by, and the sound image localization control processing step. A control information generating step for generating control information for performing signal processing, A sending information generating step of superimposing the audio signal and the control information on a signal to generate sending information, and performing signal processing on the audio signal in real time to correspond to a video based on the video signal. A sound image localization signal processing method characterized in that a sound image localization position is controlled.

About 13. The audio signal subjected to the signal processing through the control information in the production system are synthesized by the sound source data combining means is input from a plurality of sound sources, based on the video signal having the angle information with respect to the reference position In a sound image localization signal processing method for subjecting reproduction information to signal processing so as to localize a reproduction sound image in an arbitrary position in a reproduction system so as to correspond to a video, the audio signal and the audio signal superposed on the video signal in the reproduction information and Reproduction information reading step for reading the video signal, the audio signal, and the control information from the control information, a video signal reproducing step for reproducing the video signal, and angle information of a reproduced sound image of a video based on the video signal. Selection step of selecting the viewpoint selection information of the user to be selected and the selection step above Sound image localization control processing step for performing in real time signal processing for sound image localization on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image. A sound image localization signal processing method, wherein signal processing is performed to control a sound image localization position corresponding to a video image based on the video signal.

[Procedure Amendment 2]

[Document name to be amended] Statement

[Correction target item name] 0036

[Correction method] Change

[Correction content]

The parameter setting section 33 performs an equalizing processing section 33-1 for performing equalizing processing of phase equalization on the audio signal with a predetermined parameter, and a reverberation reverb processing based on the operation signal C from the track setting section 32. The reverb processing unit 33-2 for performing, the volume processing unit 33-3 for performing the volume processing, the pan processing unit 33-4 for performing the pan processing of the movement of the sound image, and the HRTF (head related transfer function: Hea).
d Related Transfer Function
The n) by changing the transfer function reaching the user's ear subjected to acoustic characteristics processing HRTF processor 33-5
And is configured.

[Procedure 3]

[Document name to be amended] Statement

[Correction target item name] 0050

[Correction method] Change

[Correction content]

With this address signal, the digitally recorded impulse response from the virtual sound source position with respect to the reference direction of the head recorded in the memory in advance to both ears, and the time difference between both ears and both ears are recorded from other memories. reading a control signal or Inpa ls e response representing the level difference between. In the convolution integrator or controller, to correct and change the this Inpa ls e response or control signals and audio signals in real time.

[Procedure amendment 4]

[Document name to be amended] Statement

[Correction target item name] 0053

[Correction method] Change

[Correction content]

[0053] from the memory, of the data corresponding to the table, between a digital recorded Inpa ls e response or both ears reaching the ears from the virtual sound source position with respect to reference direction corresponding to the orientation of the listener's perspective for Free Viewpoint Video A control signal representing the time difference between the two and the level difference between the two ears is taken out, and this data is supplied to a convolution integrator or a controller.

[Procedure Amendment 5]

[Document name to be amended] Statement

[Correction target item name] 0054

[Correction method] Change

[Correction content]

[0054] viewpoint digitally recorded Inpa ls e response leading to both ears from the virtual sound source position from the memory based on the angle information for the criterion direction of the head corresponding to the orientation of the listener's perspective to the free viewpoint video from the selecting section 5 Alternatively, a control signal representing a time difference between both ears and a level difference between both ears is taken out, and this data is supplied to a convolution integrator or a controller.

[Procedure correction 6]

[Document name to be amended] Statement

[Correction target item name] 0055

[Correction method] Change

[Correction content]

[0055] In this manner, the audio signal L supplied to the speaker, R represents a digital recorded Inpa ls e reaches the ears from the virtual sound source position with respect to reference direction corresponding to the orientation of the listener's perspective for Free Viewpoint Video the correction of the control signal representing the level difference between the time difference and binaural between response or aural performed Runode, also for the movement of the listener's perspective to the free viewpoint video, a plurality of speakers to the moved direction It is possible to obtain a sound field feeling that the speaker is placed at the virtual sound source position and is reproduced by this speaker.

[Procedure Amendment 7]

[Document name to be amended] Statement

[Correction target item name] 0065

[Correction method] Change

[Correction content]

A parameter setting section 44 is provided on the GUI application screen 41, and when the sound creator clicks the icon of the parameter setting section 44, the track setting section 32 shown in FIG.
Based on the operation signal C from the equalizing processing unit 44-1 for performing equalizing processing on the audio signal with a predetermined parameter, a reverb processing unit 44-2 for performing reverb processing, and a volume processing unit 44-3 for performing volume processing.
The track setting unit 43 sets the pan processing unit 44-4 for performing the pan processing, and the HRTF processing unit 44-5 for performing the acoustic characteristic processing by changing the transfer function reaching the user's ear by the HRTF. it can be set to the audio signal for the frame image signal processing state. Corrected audio data can be obtained according to the setting by the parameter setting unit 44.

[Procedure Amendment 8]

[Document name to be amended] Statement

[Correction target item name] 0102

[Correction method] Change

[Correction content]

In step S6, the recording mode of the time metadata is recorded.
If not, proceed to step S7, step S77
Then, the channel mapping signal processing by the operation metadata is performed.
Determine whether there is an operation of the science department. Specifically, FIG.
For example, the operation for the equalizer processing unit 33-1 shown in
High frequency band or low frequency according to the angle information of information C
Operational metade for lowering equalizer processing in band
Operation information for the data and reverb processing unit 33-2
High frequency band or low frequency band according to the angle information of C
Operational metadata for reduced reverb processing in,
For example, the angle information of the operation information C with respect to the pan processing unit 33-4.
Operation metadata for moving the position of the sound according to the information, H
For example, the angle of the operation information C with respect to the RTF processing unit 33-5
For improving or lowering frequency transfer characteristics according to information
Signal processing by operation metadataButDetermine if there is
It

[Procedure Amendment 9]

[Document name to be amended] Statement

[Correction target item name] 0129

[Correction method] Change

[Correction content]

Further, in the sound image localization signal processing device of the present invention, in the above description, the control information generating means generates the control information by the parameter corresponding to the time axis information.
Therefore, there is an effect that the sound image localization can be changed by the control information corresponding to the time information.

Claims (12)

[Claims] 1. A sound source input from a plurality of sound sources in a production system so that a reproduced sound image in a reproduction system is localized at an arbitrary position so as to correspond to a video based on a video signal having angle information with respect to a reference position. In a sound image localization signal processing device for performing signal processing on an audio signal synthesized by data synthesizing means, a selecting means for selecting viewpoint selection information of a user which is angle information of a reproduced sound image of a video based on the video signal, and the selecting means. The sound image localization control processing means for performing in real time signal processing for sound image localization on the channel of the audio signal corresponding to the viewpoint selection information of the user which is the angle information of the reproduced sound image selected by the sound image localization control processing means. Control information generating means for generating control information for performing signal processing, and the video signal and the audio signal. And a transmission information generating means for superimposing the control information to generate transmission information, and subjecting the audio signal to signal processing in real time to control the sound image localization position corresponding to the video based on the video signal. A sound image localization signal processing device characterized in that. 2. The sound image localization signal processing device according to claim 1, wherein the sound image localization control processing means applies signal processing to all channels of the audio signals of a plurality of channels by the sound image localization control processing means. A second operation mode in which the sound image localization control processing means performs signal processing, and the other channels are not subjected to the signal processing. A sound image localization signal processing device having an operation mode setting means for setting and. 3. The sound image localization signal processing device according to claim 1, wherein the sound image localization control processing means corresponds to viewpoint selection information of a user, which is angle information of the reproduced sound image selected by the selection means. Sound image localization signal processing including track setting means for generating an operation signal for performing the signal processing on the audio signal by frame processing, horizontal angle processing, zoom processing and / or vertical angle processing of the video signal. apparatus. 4. The sound image localization signal processing device according to claim 1, wherein the sound image localization control processing means, based on the operation signal from the track setting means, performs equalizing processing and reverberation on the audio signal with a predetermined parameter. A sound image localization signal processing device having a parameter setting means for performing processing, volume processing, pan processing and / or transfer characteristic processing. 5. The sound image localization signal processing device according to claim 1, wherein the control information generation means generates the control information by a parameter corresponding to the viewpoint selection information. 6. The sound image localization signal processing device according to claim 1, wherein the control information generation means generates the control information by a parameter corresponding to time axis information. 7. The sound image localization signal processing device according to claim 2, wherein the sound image localization control processing means sets the first operation mode and the second operation mode on the time axis by the operation mode setting means. A sound image localization signal processing device characterized by being switched. 8. The sound image localization signal processing device according to claim 4, wherein the sound image localization control processing means designates only a virtual surround channel of the audio signals of a plurality of channels and performs the signal processing. The sound image localization signal processing device, wherein the real surround channel is not subjected to the above signal processing. 9. An audio signal obtained by synthesizing sound source data input from a plurality of sound sources by a synthesizing means and performing signal processing by control information in a production system, based on a video signal having angle information with respect to a reference position. In a sound image localization signal processing device for performing signal processing on reproduction information so as to localize a reproduction sound image at an arbitrary position in a reproduction system so as to correspond to an image, the audio signal and the audio signal superimposed on the video signal in the reproduction information and Reproduction information reading means for reading the video signal, the audio signal and the control information from the control information, a video signal reproducing means for reproducing the video signal, and angle information of a reproduced sound image of the video based on the video signal. Selection means for selecting the viewpoint selection information of the user and the angle of the reproduced sound image selected by the selection means. Sound image localization control processing means for performing signal processing for sound image localization in real time on the channel of the audio signal corresponding to the user's viewpoint selection information to be information, and performing signal processing on the audio signal in real time A sound image localization signal processing device, characterized in that a sound image localization position is controlled corresponding to a video image based on a video signal. 10. The sound image localization signal processing device according to claim 9, wherein the sound image localization control processing means sets a parameter for performing at least surround pan processing on the audio signal according to a predetermined parameter, corresponding to the viewpoint selection information. A sound image localization signal processing device comprising means. 11. A sound source input from a plurality of sound sources in a production system so that a reproduced sound image in a reproduction system is localized at an arbitrary position so as to correspond to an image based on a video signal having angle information with respect to a reference position. In a sound image localization signal processing method for performing signal processing on an audio signal obtained by synthesizing data by a synthesizing means, a selecting step for selecting viewpoint selection information of a user which is angle information of a reproduced sound image of a video based on the video signal, and the selecting step The sound image localization control processing step of performing signal processing for sound image localization in real time on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image selected by, and the sound image localization control processing step. A control information generating step for generating control information for performing signal processing, A sending information generating step of superimposing the audio signal and the control information on a signal to generate sending information, and performing signal processing on the audio signal in real time to correspond to a video based on the video signal. A sound image localization signal processing method characterized in that a sound image localization position is controlled. 12. An audio signal obtained by synthesizing sound source data input from a plurality of sound sources by synthesizing means and performing signal processing by control information in a production system, based on a video signal having angle information with respect to a reference position. In a sound image localization signal processing method for subjecting reproduction information to signal processing so as to localize a reproduction sound image in an arbitrary position in a reproduction system so as to correspond to a video, the audio signal and the audio signal superposed on the video signal in the reproduction information and Reproduction information reading step for reading the video signal, the audio signal, and the control information from the control information, a video signal reproducing step for reproducing the video signal, and angle information of a reproduced sound image of a video based on the video signal. Selection step of selecting the viewpoint selection information of the user to be selected and the selection step above Sound image localization control processing step for performing in real time signal processing for sound image localization on the channel of the audio signal corresponding to the viewpoint selection information of the user, which is the angle information of the reproduced sound image. A sound image localization signal processing method, wherein signal processing is performed to control a sound image localization position corresponding to a video image based on the video signal.