JP5174527B2 - Acoustic signal multiplex transmission system, production apparatus and reproduction apparatus to which sound image localization acoustic meta information is added - Google Patents

Description

  The present invention relates to a technique for performing sound reproduction with a high sense of presence in reception viewing of an ultra-high-definition television system, and in particular, using a sound image localization acoustic meta information transmitted from a broadcasting station together with a video, a sound image position relative to the video. The present invention relates to an acoustic signal multiplex transmission system, a production apparatus, and a reproduction apparatus that identify sounds that require matching and that can control sound image localization in accordance with the size of the video display on the receiving side.

  When a multi-channel audio signal broadcast together with video is applied to a video / audio system (hereinafter also referred to as a reception viewing environment) used on the receiving side, the multi-channel audio signal is a plurality of audio signals arranged around the viewer. It is used to reproduce surround sound to other speakers. For example, in the case of digital surround sound reproduction, surround left (SL) arranged on the front left (FL: Front Left), front center (C: Center), front right (FR: Front Right), left rear, and left side. : Surround Left (SR), Surround Right (SR: Surround Right) placed on the right side, and Subwoofer (SW: Sub Woofer) that reproduces bass sound are known.

  In addition, as an audio system compatible with large-screen, high-definition video systems, it can reproduce more realistic sound than the 5.1-channel surround sound system. The method is also known.

  In addition, there is a three-dimensional (stereoscopic) sound system in which channels are expanded hierarchically in the height direction to enable sound image localization in the height direction (see, for example, Non-Patent Document 1). As shown in FIG. 7, in the production of a program of this ultra-high-definition / high-sense video / audio system (hereinafter referred to as an ultra-high-definition television system), a large-screen video display (for example, 7680 × 4320 pixels) and many Program production under standard production conditions with speakers. In other words, under the standard production conditions, the large-screen display 18a is forward, the low-frequency sound effect speakers LFE1 and LFE2 are excluded, and the three-dimensional upper layers U1 to U9, the intermediate layers M1 to M10, and the lower layers L1 to L3. The sound signal is produced on the 22 channels, and the program is produced so that the image position on the screen matches the position of the sound image reproduced by the speaker.

  On the other hand, on the receiving side, it is often difficult to install a video display and audio speakers under the same standard production conditions as the program production. A small size, for example, 1920 × 1080 pixels) and a predetermined number of speakers are installed (see FIG. 8).

Akio Ando, “High Realistic Acoustics”, NHK Broadcasting Technology Laboratory, [Search Date: April 20, 2008], Internet <http://www.nhk.or.jp/strl/group/ningen_joho/ningen_joho06 .html>

  For programs produced under standard production conditions, multi-channel audio signals that take into account the sound pressure level and direction of sound are provided along with video signals for a number of objects appearing in the scene as several sound sources (virtual sound sources). The In this case, in a receiving and viewing environment in which a video display with a size smaller than the standard production conditions is installed, if the provided video signal and audio signal are reproduced as they are, reproduction of sound with a high sense of presence intended by the program producer can be realized. May not be possible. Therefore, on the receiving side, according to the receiving and viewing environment, the sound image position is adjusted so as to match the size of the video display while maintaining the reproduction of sound with a high sense of presence as it is, and for the sound that needs to match the image and the sound image position. It is desired to be able to perform sound reproduction of an ultra-high-definition television system with a high sense of presence, such as controlling and reproducing the sound.

  In addition to the sound produced by actors' dialogues and objects, the program sounds that are produced will be heard as if they are pronounced from the video on the screen. In addition, there are many “effective sounds that are not directly related to the video” such as sound effects for enhancing the realism of the entire program. Therefore, if the video and audio information intended by the producer is to be reproduced more faithfully in the reception / viewing environment, it is desirable to identify these sounds on the receiving side.

  As described above, in television viewing in a conventional reception environment (for example, home), the program is produced and transmitted based on the speaker arrangement specified by the production side, and the transmitted acoustic channel signal is transmitted from the speaker assigned to each channel. Since playback is performed as it is in the reception viewing environment, it is not possible to perform coincidence control between the video and sound image positions.

  In addition, conventional sound image localization sound meta information representing information on sound image localization between sound channels is dedicated to be used only for the reduction of the number of transmission channels and the transmission bit rate, and this sound image localization sound meta information is used for video and sound images. It could not be used to perform position matching control.

  An object of the present invention is to provide high realistic sound even in a reception viewing environment in which a video display having a size smaller than the standard production conditions is installed in order to enable sound reproduction of a super high definition television system with high realistic sound. An acoustic signal multiplex transmission system, a production device, and a sound device that can control sound image localization in accordance with the size of the video display on the receiving side for sound that needs to match the position of the image and the sound image while maintaining sound reproduction. To provide a playback device.

  That is, the acoustic signal multiplex transmission system according to the present invention includes a production apparatus that multiplexes and transmits predetermined sound image localization acoustic meta information to an acoustic signal produced under standard production conditions, and receives the acoustic signal, and the sound image localization. A sound signal multiplex transmission system including a playback device that converts to a new sound signal based on the sound meta information and plays back the image, and the production device displays a video displayed on a display at the time of program production that is a standard production condition When mixing multi-channel audio signals that are synchronized with a specific number of speakers, whether the sound needs to match the sound source position with the video, or if the sound needs to match the sound image position, standard production Sound image localization acoustic meta information indicating sound attributes including the position of the sound image localization in the condition and the acoustic feature meta information related to the sound image localization is generated for each frame of the acoustic signal. Means and a means for encoding and transmitting an acoustic signal in the standard production conditions together with the sound image localization acoustic meta information, and the reproducing apparatus receives the encoded acoustic signal and the sound image localization acoustic meta from the production apparatus. Means for receiving and decoding each of the information, means for identifying whether or not the decoded acoustic signal is a sound that requires matching of the image and the sound source position for each frame based on the decoded sound image localization sound meta information, Only when the sound needs to be matched with the sound source position, the sound image localization position in the standard production condition is the sound image position suitable for the display and the predetermined number of speakers in the reception viewing environment previously associated with the standard production condition. Based on the acoustic feature meta-information relating to the sound image localization, and at the position of the sound image localization after the coordinate conversion A means for converting a sound signal to be combined into a sound signal of a predetermined channel corresponding to a speaker in the reception viewing environment for each sound source, and the converted sound signal of the predetermined channel, and a match between a video and a sound source position are required. In a receiving and viewing environment in which the sound signal of the predetermined channel that is not a sound is synthesized and a sound is reproduced from a speaker corresponding to the predetermined channel, and a display having a size different from the standard production conditions is installed. The sound image position is controlled so as to match the size of the display for the sound that needs to match the video and the sound image position.

  In the acoustic signal multiplex transmission system according to the present invention, the sound image localization acoustic meta information includes, as the acoustic feature meta information, “transmission channel / acoustic main frequency component”, “acoustic frequency contour”, “interchannel level difference (ICLD)”. : Inter-Channel Level Differentials) and “Inter-Channel Time Differences (ICTD)” information, and means for converting into an audio signal of a predetermined channel corresponding to a speaker in the reception viewing environment, Using the information of the combination of the “transmission channel / acoustic main frequency component”, “acoustic frequency contour”, “ICTD”, and “ICLD”, the sound matching sound source signal for each channel is determined. .

In the acoustic signal multiplex transmission system according to the present invention, the sound image localization acoustic meta information includes, as the acoustic feature meta information, “acoustic frequency contour”, “sound image position”, and “inter-channel correlation (ICC)”. ) "Information,
The means for converting to an acoustic signal of a predetermined channel corresponding to a speaker in the reception viewing environment uses the information of the combination of the “acoustic frequency contour”, “sound image position”, and “ICC” to output the sound for each channel. The coincidence sound source signal is determined.

  In the acoustic signal multiplex transmission system according to the present invention, the sound image localization acoustic meta information includes, as the acoustic feature meta information, “acoustic frequency contour”, “transmission channel number information”, and “inter-channel correlation (ICC: Inter-)”. Means for converting into an audio signal of a predetermined channel corresponding to a speaker in the reception viewing environment, the “acoustic frequency contour”, “sound image position”, “transmission channel number information”, and Using the information on the combination of “ICC”, the sound matching sound source signal for each channel is determined.

  Furthermore, the production apparatus according to the present invention is a production apparatus that multiplexes and transmits predetermined sound image localization acoustic meta information to an acoustic signal produced under standard production conditions, and is used as a display at the time of program production as standard production conditions. When mixing multi-channel audio signals that are synchronized with the displayed video for a certain number of speakers, whether the sound needs to match the position of the video and the sound source, or if the sound needs to match the position of the video and the sound image Includes means for generating sound image localization acoustic meta information indicating sound attributes in units of frames of the acoustic signal, including sound image localization positions in standard production conditions and acoustic feature meta information related to the sound image localization; And means for encoding and transmitting an acoustic signal under production conditions together with the sound image localization acoustic meta information.

  Furthermore, the playback device according to the present invention receives a sound signal produced according to a predetermined standard production condition, converts the sound signal to a new sound signal based on the sound image localization sound meta information attached to the sound signal, and plays back the sound signal. A device that receives and decodes an encoded acoustic signal and sound image localization acoustic meta information, and, based on the decoded sound image localization acoustic meta information, the image and sound source position of the decoded acoustic signal in units of frames; Means for discriminating whether or not the sound needs to be matched, and a display and a predetermined number of speakers in the receiving and viewing environment previously associated with the standard production conditions only when the sound and the sound source position need to be matched Based on the sound feature meta information related to the sound image localization, and means for coordinate conversion of the sound image position conforming to the sound image localization from the information of the position of the sound image localization in the standard production conditions Means for converting an acoustic signal that conforms to the position of the sound image localization having undergone the coordinate conversion into an acoustic signal of a predetermined channel corresponding to a speaker in the reception viewing environment for each sound source; and the acoustic signal of the converted predetermined channel; And means for synthesizing an image and an acoustic signal of the predetermined channel that does not require sound source position matching and reproducing sound from a speaker corresponding to the predetermined channel.

  As a result, for the sound that needs to match the image and the sound image position, it is possible to control the sound image position appropriately so as to match the image display.

  According to the present invention, even in a reception viewing environment in which a video display having a size smaller than the standard production conditions is installed, while maintaining the reproduction of sound with a high sense of presence, and for the sound that requires matching between the image and the sound image position, By controlling and reproducing the position of the sound image so as to match the video display, it is possible to reproduce the sound of an ultra-high-definition, high-sense TV with high presence.

  An acoustic signal multiplex transmission system according to an embodiment of the present invention will be described below. The acoustic signal multiplex transmission system according to the embodiment includes the production apparatus and the reproduction apparatus according to an embodiment of the present invention.

[System configuration]
The acoustic signal multiplex transmission system of the present embodiment includes a production apparatus that multiplexes and transmits predetermined sound image localization acoustic meta information to an acoustic signal produced under standard production conditions, and receives the acoustic signal, and the sound image localization acoustic meta information. And a playback device for converting to a new sound signal and playing it back. In the audio signal multiplex transmission system of this embodiment, even in a reception viewing environment in which a video display having a size smaller than the standard production conditions is installed, the sound that needs to match the position of the video and the sound image should be matched to the size of the video display. This is a system for controlling the sound image position and reproducing the sound of a super-high-definition, high-realistic television with high presence.

  The production apparatus according to the present embodiment is an apparatus that multiplexes and transmits an acoustic signal to which sound image localization acoustic meta information is added, and encodes and multiplexes the acoustic signal and the sound image localization acoustic meta information into one bit stream. The multiplexed / acoustic signal to which the “sound image localization acoustic meta information” is added is transmitted to a remote place such as a home via radio waves or an IP line.

  That is, the production apparatus of the present embodiment, when mixing a multi-channel audio signal synchronized with the video displayed on the display at the time of program production, which is a standard production condition, for a predetermined number of speakers, Indicates whether or not the sound needs to be matched, and in the case of a sound that requires matching between the image and the sound image position, the sound image localization including the position of the sound image localization in the standard production conditions and the acoustic feature meta information related to the sound image localization Sound image localization acoustic meta information is generated for each frame of the acoustic signal, and the acoustic signal under standard production conditions is encoded and transmitted together with the sound image localization acoustic meta information.

  At the time of production, when producing and recording program sound, whether or not the sound needs to match the position of the video and the sound source, and if the sound needs to match the position of the video and the sound image, Sound image localization acoustic meta information indicating sound attributes such as a position of sound image localization (or a position on the screen) and acoustic feature meta information related to localization is created. On the other hand, the sound that does not require the coincidence between the video and the sound image position can be identified by the sound image localization acoustic meta information.

  The playback device of this embodiment receives an audio signal to which sound image localization acoustic meta information is added, and it is necessary to match the position of the image and the sound image in accordance with the reception viewing environment in which an image display having a size smaller than the standard production conditions is installed. To control the sound image position suitable for the video display.

  That is, the playback apparatus of the present embodiment receives the encoded acoustic signal and the sound image localization acoustic meta information from the production apparatus, respectively decodes the decoded acoustic signal based on the decoded sound image localization acoustic meta information. Identify whether or not the sound needs to match the position of the video and the sound source in units of frames, and only when the sound needs to match the position of the video and the sound source A sound image position suitable for a predetermined number of speakers is determined by coordinate conversion from information on the position of sound image localization in standard production conditions (in addition, conversion of XYZ coordinates may be conversion using sin / cos), Based on the acoustic feature meta information related to the sound image localization, an acoustic signal suitable for the position of the coordinated sound image localization corresponds to the speaker in the reception viewing environment for each sound source. Converts the sound signal of a predetermined channel into a predetermined channel, and combines the converted sound signal of the predetermined channel with the sound signal of the predetermined channel that does not require sound and the position of the sound source to match. Play sound from the speaker you want to play.

  As will be described later, in the reception viewing environment in which a video display having a size smaller than the standard production conditions is installed, the playback device of the present embodiment, for the sound that requires matching between the video and the sound image position, Based on the separated "sound image localization acoustic meta information", examples of "extraction of main acoustic frequency components of projection matching sound source" and "synthesis of main matching frequency components and acoustic frequency contours of projection matching source" Processing is performed to control the position of the sound image to match the size of the video display. Furthermore, the playback apparatus of the present embodiment does not perform any special processing for the sound that does not require the coincidence of the video and the sound image position, and performs the same processing as the conventional one. The output signal for each speaker generated in accordance with the reception / viewing environment is combined with the case where the video and sound image positions coincide with each other, that is, combined and reproduced from each speaker.

  FIG. 1 is a diagram illustrating a production apparatus that performs multiplex transmission by attaching sound image localization acoustic meta information to an acoustic signal to be transmitted. Note that, for example, in digital broadcasting, it is known that multiplexed audio signals are multiplexed and transmitted in synchronization with a target video signal. FIG. 1 shows only the multiplexed audio signals. The production apparatus includes an acoustic recording / playback apparatus 1, an acoustic mixing console 2, an acoustic signal production transmission channel conversion apparatus 3, an acoustic signal encoding apparatus 4, an acoustic localization acoustic meta information multiplexing apparatus 5, and an acoustic signal / meta information. A multiplexing device 6, an acoustic mixing control musical score data generation device 7, and a microphone 8 that collects speech in accordance with the scene are provided.

  The sound recording / playback apparatus 1 records predetermined sound according to a control signal (acoustic signal generation / end control signal) instructing generation or termination of an acoustic signal based on the musical score data generated by the acoustic mixing control musical score data generation apparatus 7. Alternatively, the recorded multi-channel acoustic signals are reproduced and output to the acoustic mixing console 2.

  The acoustic mixing table 2 includes a dialogue signal recorded via the microphone 8 in accordance with a control signal (acoustic mixing control signal) instructing mixing of the acoustic signal based on the musical score data generated by the acoustic mixing control musical score data generation device 7. Then, the sound signal reproduced from the sound recording / playback apparatus 1 is mixed, and the produced sound signal for each of a plurality of channels produced under the standard production conditions described above is output to the sound signal production / transmission channel converter 3.

  The acoustic signal production transmission channel conversion device 3 converts the produced acoustic signal into an acoustic signal for each frame for the transmission channel, and outputs the converted acoustic signal for transmission (transmission acoustic signal) to the acoustic signal encoding device 4. To do.

  The acoustic signal encoding device 4 encodes the transmission acoustic signal according to a predetermined acoustic encoding method, and outputs the encoded transmission acoustic signal in frame units to the acoustic signal / meta information multiplexing device 6.

  The sound localization sound meta information multiplexing device 5 uses the sound signal from the sound recording / playback device 1 and the produced sound signal from the sound mixing table 2 to identify whether the sound needs to match the sound source position. In the case of sound that needs to match the position of the image and the sound image, the sound image localization acoustic meta that indicates the sound attributes such as the position of the sound image localization (or position on the screen) in the standard production conditions and the acoustic feature meta information related to the localization. Information is created in units of frames (described later), and the sound image localization acoustic meta information is encoded according to a predetermined acoustic encoding method and transmitted to the acoustic signal / meta information multiplexing device 6.

  The acoustic signal / meta information multiplexing device 6 multiplexes the encoded transmission acoustic signal for each frame and the encoded acoustic localization acoustic meta information output from the acoustic localization acoustic meta information multiplexing device 5 for each frame. Alternatively, the data is output to the playback device via an IP line or the like.

  The acoustic mixing control musical score data generation device 7 generates musical score data by an operation of a sound production engineer, generates an acoustic signal generation / termination control signal to the acoustic recording / playback device 1, and an acoustic mixing control signal to the acoustic mixing console 2. Is output.

  As described above, the production apparatus according to the present embodiment determines whether the sound needs to match the video and sound image positions based on the intention of the program producer, that is, information operated by the sound production engineer, and the video and sound image positions. If the sound needs to match, the program sound is produced and recorded with information indicating sound attributes, such as the sound image localization position information in the standard production conditions and the sound image localization acoustic meta information related to the sound image localization. The signal and the sound image localization acoustic meta information are encoded, multiplexed into one bit stream, and transmitted.

  FIG. 2 is a diagram illustrating a playback apparatus that converts sound into sound signals for reception environment using sound image localization sound meta information and plays back the sound signals. Note that it is known that an audio signal to be reproduced is reproduced in synchronization with a target video signal in, for example, digital broadcasting, and FIG. 2 shows only the audio signal to be reproduced. The reproduction apparatus of the present embodiment includes an acoustic signal / meta information demultiplexing device 11, an acoustic localization acoustic meta information separation device 12, an acoustic signal decoding device 13, an acoustic signal localization converter 14, and an acoustic signal localization conversion control unit. 15, an acoustic signal reproduction device 16, and a speaker set 17.

  The acoustic signal / meta information demultiplexing device 11 receives the transmission acoustic signal and acoustic localization acoustic meta information multiplexed and encoded in units of frames from the production device, and encodes the transmission acoustic signal and acoustic localization sound. The meta information is separated and sent to the acoustic signal decoding device 13 and the acoustic localization acoustic meta information separation device 12, respectively.

  The sound localization sound meta information separation device 12 monitors the presence or absence of sound image localization sound meta information in units of frames, and decodes the encoded sound localization sound meta information when there is sound image localization sound meta information. The meta information for each item is separated and sent to the acoustic signal localization conversion control unit 15.

  The acoustic signal decoding device 13 decodes the encoded transmission acoustic signal and sends the acoustic signal for each of a plurality of channels at the time of production to the acoustic signal localization converter 14.

  The acoustic signal localization converter 14 converts an acoustic signal for each of a plurality of channels at the time of production into an acoustic signal for each of a plurality of channels according to the reception viewing environment by using the control signal of the acoustic signal localization conversion control unit 15. It is sent to the signal reproduction device 16.

  The acoustic signal localization conversion control unit 15 analyzes the acoustic localization acoustic meta information separated for each item received from the acoustic localization acoustic meta information separation device 12, and outputs the acoustic signal at the time of production of the corresponding frame to the video display. A control signal for converting into an acoustic signal for each of a plurality of channels according to the reception viewing environment including the size is sent to the acoustic signal localization converter 14. This conversion control will be described later in detail. The acoustic signal localization conversion control unit 15 arbitrarily selects and sets the result of analyzing the meta information separated for each item so as to correspond to the speaker according to the position coordinates in the reception viewing environment. It is possible.

  The acoustic signal reproducing device 16 sends out acoustic signals for a plurality of channels corresponding to the reception viewing environment to the respective speakers in the speaker set 17 as reproduced signals.

  The speaker set 17 reproduces a reproduction signal corresponding to each of the sound signals for each of a plurality of channels in the reception / viewing environment by the plurality of speakers Sp1 to Sp7. The speaker set 17 shown in FIG. 2 shows only seven speakers, but is merely an example. For example, a 5.1 channel speaker set, a front 5 channel + side / rear 5 channel sound system is used. A speaker set or a three-dimensional (three-dimensional) sound system speaker set can be used, and among these speakers, a speaker for sound image localization conversion is preferably arranged at a position coordinate corresponding to the acoustic localization acoustic meta information. It is. That is, on the playback device side, position coordinates corresponding to the acoustic localization acoustic meta information on the production device side are set in advance in the acoustic signal localization conversion control unit 15.

  As described above, the playback device uses the sound image localization acoustic meta information to convert into an acoustic signal corresponding to the reception viewing environment and reproduces it. In other words, the sound image localization acoustic meta information is used for the “projection matching sound source” that requires matching between the image and the sound image position intended on the production side in order to maintain the sound reproduction with high presence in the reception viewing environment. Then, an audio channel signal is generated by converting the sound image position on the production side to an appropriate sound image position so as to match the size of the video display, and is reproduced from each speaker.

  Hereinafter, the process of the sound image position conversion control in the acoustic signal localization conversion control unit 15 will be described.

  FIG. 3 shows an example of speaker arrangement in the intermediate layer in a reception viewing environment on a display smaller than the standard production conditions. Here, it is assumed that the reception / viewing environment is configured with the same speaker arrangement as in the standard production conditions, and in the reception / viewing environment, video is projected on the display 18b having a size smaller than the display 18a, which is the standard production conditions.

  In other words, even if the speaker arrangement in the reception viewing environment is the same as in the standard production conditions, if the display is smaller than the standard production conditions, the sound source (virtual sound source) may be adapted to many objects appearing in the scene. Since a multi-channel audio signal that takes into account the sound pressure level and sound direction is provided together with the video signal, sound image position conversion control is required when the production side provides the sound coincidence.

  Therefore, at the time of production, in accordance with the production of normal program sound, based on the intention of the program producer and information operated by the sound production engineer, sound image localization sound information indicating sound attributes is attached and the program sound is recorded. Production and recording.

  Here, the sound image localization acoustic meta information associated with the “projection sound source” will be described. On the transmission side, the sound image localization acoustic meta information on the production side is recorded so as to be transmitted for each frame obtained by dividing the acoustic time sample value into fixed time units. On the receiving side, this sound image localization acoustic meta-information is extracted for each frame, and the video and audio are subjected to calculations for adapting to the video / acoustic system in the receiving environment on the receiving side and reproduced. On the transmission side, for each frame representing the acoustic time sample, a flag indicating the presence / absence of meta information (for example, 1 if there is meta information, 0 if there is no meta information) is attached, and this flag is monitored on the receiving side. Then, the sound image localization acoustic meta information of the corresponding frame is extracted.

[Sound image localization acoustic meta information]
The sound image localization sound meta information includes at least a “use sound signal channel number” and a “sound matching sound source flag”, a “sound source continuous flag”, “sound image position”, “transmission channel number information”, “acoustic frequency contour”, “Transmission channel / acoustic main frequency component”, “Inter-Channel Level Differences (ICLD)”, “Inter-Channel Time Differences (ICTD)”, and “Inter-Channel Correlation (ICC)” -Channel Coherence) ", at least one or more information items necessary on the receiving side can be selectively included. The “acoustic frequency contour”, “transmission channel / acoustic main frequency component”, “inter-channel level difference (ICLD)”, “inter-channel time difference (ICTD)”, and “inter-channel correlation (ICC)” It is information created for each critical bandwidth or for every ½ critical bandwidth of hearing, and is positioned as acoustic feature meta-information created for each projection sound matching sound source.

  “Used audio signal channel number” indicates the channel number of the audio signal used for the program sound of program production (not silence). Even in the case of a speaker arrangement in a reception / viewing environment different from the standard production conditions, the playback apparatus can convert the channel number associated with the world coordinates in the reception / viewing environment based on this information.

  The “projection matching sound source flag” indicates the presence or absence of a sound source that requires matching between the video and the sound image position in order to identify on the receiving side a sound source that matches the video and the sound image position as information at the time of program production. This flag is set to 1 at the start of the coincidence sound source and 0 at the end, for example.

  The “sound source continuous flag” indicates that a sound source having a number that requires matching between a video and a sound image position (hereinafter referred to as a “sound matching sound source”) is continuous between adjacent frames. It can be used as reference information when discriminating the “sound matching sound source flag”. For example, it is preferable to use this sound source continuation flag when correcting the continuity of sound between adjacent frames on the receiving side using a filter that makes smoother. This flag is set to 1 when, for example, it is continuous between adjacent frames, and set to 0 at the end.

  “Sound image position” indicates the position of the sound image of the program production as one of the acoustic feature meta information created for each sound matching sound source. For example, the coordinate information of the world coordinates X, Y, and Z in the program production environment (including the screen size, the arrangement for each type of speaker, and the set viewing position) used on the producer side can be used.

  “Transmission channel number information” indicates a speaker channel number through which an acoustic signal is transmitted as one piece of acoustic feature meta information created for each sound matching sound source. When there is a sound image position between the speakers, it indicates an acoustic channel surrounding the screen at the time of production. This information is information to be transmitted for reference and / or processing convenience, although the transmission channel can generally be specified by the information of “sound image position”. For example, if this transmission channel number information is used on the receiving side, the speaker channel number can be immediately identified, so that the channel assignment processing speed can be improved.

  The “acoustic frequency contour” indicates level information for each frequency of the sound source as one of the acoustic feature meta information created for each sound matching sound source.

  “Transmission channel / acoustic main frequency component” indicates the main frequency component of the sound matching sound source in the channel through which the sound signal is transmitted as one of the acoustic feature meta information created for each sound matching sound source.

  “Inter-channel level difference (ICLD)” indicates a level difference between channels through which an acoustic signal is transmitted.

  “Interchannel time difference (ICTD)” indicates a time difference between channels through which an acoustic signal is transmitted as one piece of acoustic feature meta information created for each sound matching sound source.

  “Inter-channel correlation (ICC)” indicates correlation between channels through which an acoustic signal is transmitted as one piece of acoustic feature meta information created for each sound matching sound source. For example, when there are channels numbered from 0 to 24, the ratio of the signal level of the adjacent channel of the sound image generated between the adjacent channels is represented.

  Note that the “level difference between channels (ICLD)”, “time difference between channels (ICTD)” and “correlation between channels (ICC)” should be specified on the playback device side based on the information of “used acoustic signal channel number”. However, it should be noted that the information is effective when the processing load on the playback device side is large, and is not necessarily required. However, it is preferable to transmit on the transmission side in consideration of the processing cost of the playback device.

  Next, an acoustic channel signal processing method performed by the playback apparatus will be described.

[Processing method of playback device]
FIG. 4 shows a flowchart of an acoustic channel signal processing method performed by the playback apparatus. In the reception / viewing environment, the playback device plays back sound with a high sense of presence while controlling the sound image position according to the size of the display for sounds that require matching between the image and the sound image position.

  In the description of the present embodiment, the sound source that requires the coincidence of the image and the sound image position is referred to as “projection sound source”, and the sound source that does not need to be referred to as “effect sound source”.

  In order to describe the sound image position control processing method in the playback device, a case where a sound image in a horizontal plane is moved to the video display position will be described. Note that it is possible to move the sound image in the vertical (up and down) direction by a similar method.

  In step S1, the sound localization sound meta information separation device 12 of the playback apparatus monitors the presence or absence of sound image localization sound meta information in units of frames. For example, if the flag value is “1”, the sound image localization acoustic meta information is attached to identify the frame to be processed. If the flag value is “0”, the sound image localization acoustic meta information is not attached. The frame is identified as a frame that does not require processing. If there is sound image localization acoustic meta information, the encoded acoustic localization acoustic meta information is decoded and separated into meta information for each item, and the process proceeds to step S2.

  In step S2, the sound signal localization conversion control unit 15 of the playback device specifies the channel of the sound to be used with the information item “used sound signal channel number” and matches the position of the image and the sound image with the information item “projection matching sound source flag”. Identify the presence or absence of a sound source that requires. When the sound source is identified as an “effect sound source”, the acoustic signal localization converter 14 is controlled so as to output an acoustic signal corresponding to the specified channel of the used sound without performing the acoustic signal localization conversion. On the other hand, when it is identified as “projection sound source,” the process proceeds to step S3 in order to control the acoustic signal localization converter 14 to perform acoustic signal localization conversion.

  In step S3, if the sound signal localization conversion control unit 15 of the playback apparatus identifies that the sound source is a "sound matching sound source", the sound image position at the time of production based on the information item "sound image position" or "transmission channel number information" And the main acoustic frequency component of the sound matching sound source is extracted from the information item “transmission channel / main acoustic frequency component”.

  In step S4, the acoustic signal localization conversion control unit 15 of the playback device synthesizes the main acoustic frequency component of the extracted sound matching sound source and the level information for each frequency of the sound source extracted from the information item “acoustic frequency contour”. , “Generate sound projection matching sound source”.

  If it is determined in step S5 that the number of sound sources that require matching between the video and sound image positions has been completed, the process proceeds to step S6. If there are remaining sound sources, the process returns to step S3.

  In step S 6, the generated “projection matching sound source” is synthesized by the sound signal localization conversion control unit 15 of the playback apparatus with the “effect sound source” that is a sound that does not require matching between the image and the sound image position. The acoustic signal localization converter 14 is controlled.

  In step S7, the acoustic signal localization converter 14 outputs the synthesized sound as a channel acoustic signal output from each speaker via the acoustic signal reproducing device 16.

  In addition, an example of the method of “extraction of projection matching sound source” in step S3 and “generation of projection matching sound source” in step S4 will be described.

  As a sound image extraction method, the information item “Transmission channel / acoustic main frequency component” is used in the sound image localization sound meta information of the sound matching sound source, and the frequency band indicated by this is zeroed or suppressed to suppress the original position. Extracts sound image components.

  As a method of generating a sound matching sound source, the suppression of the frequency band of the sound matching sound source is less than or equal to the signal level of the frequency band adjacent to the sound main frequency component, thereby reducing the sound main frequency component of the sound matching sound source. Perform signal suppression.

  Therefore, as a sound image generation method that matches the size of the display in the reception viewing environment, first, among the sound image localization acoustic meta information of the sound matching sound source, "transmission channel / acoustic main frequency component", "acoustic frequency contour", By using a combination of “ICTD” and “ICLD” or a combination of “acoustic frequency contour”, “sound image position”, and “ICC”, the sound matching sound source signal for each channel is determined.

  Next, a new sound image position in the reception viewing environment is obtained. The speakers and the display in the reception viewing environment are optionally set in the world signal X, Y, Z in the acoustic signal localization conversion control unit 15 on the playback device side. That is, the world coordinates X, Y, Z on the production side and the world coordinates X, Y, Z on the playback device side are determined in advance to correspond to each other.

For example, as shown in FIG. 5 (a), signal levels represented as sound image localization functions of the sound image localization functions 100M1 to 100M5 by the speakers M1 to M5 of the intermediate layer positioned around the display 18a at the time of production. It is assumed that the audio signal is output and the “projection sound source” P ₁ and P ₂ are formed, and the “sound source” E ₁ is intentionally produced. Since the display 18b in the reception viewing environment is smaller than the display 18a at the time of production, acoustic signals having signal levels represented as sound image localization functions of the sound image localization functions 100M3 to 100M5 are respectively obtained by the speakers M3 to M5 in the intermediate layer. The “sound matching sound source” P ₁ ′, P ₂ ′, which is a new sound image position, is formed, and the “sound effect sound source” E ₁ ′ is identified as information that does not require matching between the video and the sound source. It formed as the same sound source and sound effects sound source "E _1. FIG. 5 shows an example of moving a sound image in the horizontal plane.

Actually, “projection matching sound source” P ₁ ′, P ₂ ′ which is a new sound image position in the reception viewing environment mainly based on the information item “sound image position” or “transmission channel / acoustic main frequency component”. And the signal level for each frequency of the sound matching sound source output from each of the speakers 100M3 to 100M5 shown in FIG. 5 is calculated.

  There are several examples of calculating the signal level for each frequency of the projection matching sound source.

(Example 1)
Of the sound image localization sound meta information of the sound matching sound source in the channel through which the sound signal is transmitted, the channel based on “acoustic frequency contour”, “ICTD”, “ICLD”, and “transmission channel / acoustic main frequency component” The signal of the sound matching sound source for each is determined. Specifically, for example, the frequency component of the sound matching sound source of each channel is first obtained by multiplying the “acoustic frequency contour” of the sound matching sound source for each frequency component of the channel through which the acoustic signal is transmitted. Next, ICTD and ICLD for each frequency between channels are calculated, and the frequency level is corrected by comparison with “ICTD and ICLD” of the sound image localization acoustic meta information. As a result, the sound matching sound source signal for each channel is determined. In this case, the sound image localization position at the time of production can be reliably converted into the sound image localization position in the reception environment for each frame.

(Example 2)
The sound matching sound source signal is determined based on “acoustic frequency contour”, “sound image position”, and “ICC” in the sound image localization sound meta information of the sound matching sound source in the channel through which the sound signal is transmitted. Specifically, for example, the frequency component of the sound matching sound source of each channel is first obtained by multiplying the “acoustic frequency contour” of the sound matching sound source for each frequency component of the channel through which the acoustic signal is transmitted. Next, the transmission channel number is identified from the “sound image position” information, the mixing ratio of each channel is calculated based on the “sound image position” information and the transmission channel number, and the mixing ratio of each channel of the sound matching sound source is calculated. Thus, the signal of the projection matching sound source is determined. Further, the frequency level is corrected by comparing this determination signal with “ICC” of the sound image localization acoustic meta information. As a result, the sound matching sound source signal for each channel is determined. In this case, it is only necessary to sequentially calculate each frame unit based on the previous acoustic signal, and the processing burden on the playback device side can be reduced.

(Example 3)
Based on the “acoustic frequency contour”, “sound image position”, “transmission channel number information”, and “ICC” among the sound image localization sound meta information of the sound matching sound source in the channel where the sound signal is transmitted, the sound matching Determine the sound source signal. Specifically, for example, the frequency component of the sound matching sound source of each channel is first obtained by multiplying the “acoustic frequency contour” of the sound matching sound source for each frequency component of the channel through which the acoustic signal is transmitted. Next, the transmission channel number information is used without specifying the transmission channel number from the “sound image position” information. Next, the mixing ratio of each channel is calculated based on the “sound image position” information and the transmission channel number information, and the signal of the sound matching sound source is determined based on the mixing ratio of each channel of the sound matching sound source. Further, the frequency level is corrected by comparing this determination signal with “ICC” of the sound image localization acoustic meta information. As a result, the sound matching sound source signal for each channel is determined. In this case, it is only necessary to sequentially calculate each frame unit based on the previous acoustic signal, and the processing load on the playback device side can be further reduced.

  From these three examples, it can be seen that there are other combinations that realize sound image localization according to the reception viewing environment using sound image localization acoustic meta information. On the other hand, in order to reproduce the sound image position intended on the production side in the reception viewing environment, this is an example realized by using some of the sound image localization acoustic meta information. However, for example, among these sound image localization acoustic meta information, if there is an acoustic signal that is not required in the reception viewing environment (for example, a speaker corresponding to the production side is not prepared on the playback device side) It can also be set so that the reproduction apparatus can reconstruct the sound signal of a desired channel as desired. Therefore, the reproduction apparatus side may selectively use only the sound image localization acoustic meta information required in association with this.

  The sound matching sound source signal for each channel thus determined is output from each speaker as an acoustic signal having a new signal level. For example, referring to FIG. 5, the sound of the signal level expressed as the sound image localization function of the sound image localization functions 100M1 to 100M5 output from the speakers M1 to M5 of the intermediate layer positioned around the display 18a at the time of production. The signal is output as an acoustic signal of a new signal level by the speakers M3 to M5 in the intermediate layer so as to move the sound image in the horizontal plane.

Although only the speaker of a specific channel has been described in FIG. 5, the standard production conditions at the time of production are as follows. As shown in FIG. 7, the low-frequency sound effect speakers LFE ₁ and LFE ₂ are placed in front of the large screen display 18a. Except for the production with 22 channels of the three-dimensional upper layers U1 to U9, the intermediate layers M1 to M10, and the lower layers L1 to L3, for example, a display smaller than the display 18a in the reception viewing environment as shown in FIG. The acoustic signals of 22 channels composed of the same world coordinates as at the time of production together with 18b are subjected to sound image localization conversion in units of frames (see FIG. 6).

  Thereby, according to the acoustic signal multiplex transmission system of the present embodiment, which is composed of the production device and the playback device of the present embodiment, in the case where the difference between the video display / sound speaker conditions at the time of program production and the reception viewing environment is large. However, the degree of the projection sound matching sound source is suitably adapted, and it is possible to reproduce a sound with a higher presence.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. For example, when the opening angle of the central three speakers in the reception viewing environment is different from the opening angle (eg, 60 degrees) at the time of program production, the left and right speakers are installed on both sides of the video display. It is also possible to achieve high-quality sound reproduction approximately by the same processing. That is, if the speaker is not configured with the same world coordinates as at the time of production in the reception / viewing environment, the speaker and sound signals to be output in the reception / viewing environment are determined from the information on the speaker arrangement at the time of production (“sound image position”). It will be apparent to those skilled in the art that the transformation can be transformed according to world coordinates. In the above-described embodiment, the description has been made assuming that the acoustic signal having a fixed frame length is handled. However, a variable frame length may be used. In this case, additional information that can identify each frame length is sent in advance or added to each frame header. Accordingly, the invention should not be construed as limited by the embodiments described above, but only by the claims.

  The present invention is useful for a video / audio signal multiplex transmission system because sound reproduction of an ultra-high-definition television system with high presence can be performed.

It is a figure which shows the production apparatus in the acoustic signal multiplex transmission system accompanied by the sound image localization acoustic meta information of the Example by this invention. It is a figure which shows the reproducing | regenerating apparatus in the acoustic signal multiplex transmission system accompanied by the sound image localization acoustic meta information of the Example by this invention. It is a figure which shows the speaker arrangement | positioning example of the intermediate | middle layer in the reception viewing environment with the display smaller than the standard production conditions of the Example by this invention. It is a flowchart of the processing method of the acoustic channel signal performed with the reproducing | regenerating apparatus of the Example by this invention. It is the schematic of sound image localization conversion based on the sound image localization function of the Example by this invention. It is the schematic of the acoustic signal of the frame unit of the Example by this invention. It is a figure which shows the example of arrangement | positioning of the display and speaker in the standard production conditions at the time of production. It is a figure which shows the example of arrangement | positioning of the display and speaker in a reception viewing environment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Sound recording / reproducing apparatus 2 Acoustic mixing table 3 Acoustic signal production transmission channel converter 4 Acoustic signal encoding apparatus 5 Acoustic localization acoustic meta information multiplexing apparatus 6 Acoustic signal / meta information multiplexing apparatus 7 Acoustic mixing control musical score data generating apparatus 8 Microphone 11 Acoustic signal / meta information demultiplexing device 12 Acoustic localization acoustic meta information separating device 13 Acoustic signal decoding device 14 Acoustic signal localization converter 15 Acoustic signal localization conversion control unit 16 Acoustic signal reproduction device 17 Speaker set 18a Display 18b at the time of production Reception environment LFE ₁ , LFE ₂ Low-frequency sound effect speakers U _{1 to} U 9 Upper layer speakers M 1 to M 10 Middle layer speakers L _{1 to} L 3 Lower layer speakers 100 M 3 to 100 M 5 Sound image localization functions P ₁ and P ₂ Sound image position of "sound matching sound source" ₁ sound image position P ₁ of the "effect sound source" of the receiving environment ', P _2' sound image position of "sound effects sound source" of the sound image position E ₁ 'reception environment of "Utsuoto match sound source" of the receiving environment

Claims (6)

A production device that multiplexes and transmits predetermined sound image localization acoustic meta information to an acoustic signal produced under standard production conditions, and receives the acoustic signal and converts it to a new acoustic signal based on the sound localization acoustic meta information And an audio signal multiplex transmission system comprising a reproducing device for reproducing,
The production device
When mixing multi-channel audio signals synchronized with the video displayed on the display during program production, which is the standard production condition, for a predetermined number of speakers, whether the video and sound must match the sound source position Sound image localization acoustic meta information indicating sound attributes, including sound image localization positions in standard production conditions and acoustic feature meta information related to the sound image localization in the case of a sound that needs to match the sound image position with the sound signal Means for generating a frame unit of
Means for encoding and transmitting the sound signal in the standard production conditions together with the sound image localization sound meta information,
The playback device
Means for receiving and decoding the encoded acoustic signal and the sound image localization acoustic meta information from the production device;
Based on the decoded sound image localization acoustic meta-information, means for identifying whether or not the decoded sound signal is a sound that requires matching between the video and the sound source position in units of frames;
Only when the sound needs to match the position of the image and the sound source, the sound image position suitable for the display and a predetermined number of speakers in the reception viewing environment previously associated with the standard production condition is set as the sound image localization in the standard production condition. Means for converting and determining coordinates from position information;
Based on the acoustic feature meta-information related to the sound image localization, an acoustic signal suitable for the coordinate-converted sound image localization position is converted into an acoustic signal of a predetermined channel corresponding to a speaker in the reception viewing environment for each sound source. Means,
Means for synthesizing the converted sound signal of the predetermined channel and the sound signal of the predetermined channel that is not a sound that requires matching between the image and the sound source position, and reproducing sound from a speaker corresponding to the predetermined channel; Have
In the reception viewing environment where a display of a size different from the standard production conditions is installed, the sound image position is controlled so as to match the size of the display for the sound that needs to match the position of the image and the sound image, Acoustic signal multiplex transmission system. The sound image localization acoustic meta information includes, as the acoustic feature meta information, “transmission channel / acoustic main frequency component”, “acoustic frequency contour”, “inter-channel level differences (ICLD)”, and “channel”. Information on the time difference (ICTD: Inter-Channel Time Differences)
The means for converting into an acoustic signal of a predetermined channel corresponding to a speaker in the reception viewing environment is information on a combination of the “transmission channel / acoustic main frequency component”, “acoustic frequency contour”, “ICTD”, and “ICLD”. The sound signal multiplex transmission system according to claim 1, wherein a signal of a sound matching sound source for each channel is determined by using. The sound image localization acoustic meta information includes information on “acoustic frequency contour”, “sound image position”, and “inter-channel correlation (ICC)” as the acoustic feature meta information,
The means for converting to an acoustic signal of a predetermined channel corresponding to a speaker in the reception viewing environment uses the information of the combination of the “acoustic frequency contour”, “sound image position”, and “ICC” to output the sound for each channel. The acoustic signal multiplex transmission system according to claim 1, wherein a signal of a coincidence sound source is determined. The sound image localization acoustic meta information includes information on “acoustic frequency contour”, “sound image position”, “transmission channel number information”, and “inter-channel correlation (ICC)” as the acoustic feature meta information. Including
The means for converting into an acoustic signal of a predetermined channel corresponding to a speaker in the reception viewing environment uses information of a combination of the “acoustic frequency contour”, “sound image position”, “transmission channel number information”, and “ICC”. The sound signal multiplex transmission system according to claim 1, wherein a signal of a sound matching sound source for each channel is determined. A production device that multiplexes and transmits predetermined sound image localization acoustic meta information to an acoustic signal produced under standard production conditions,
When mixing multi-channel audio signals synchronized with the video displayed on the display during program production, which is the standard production condition, for a predetermined number of speakers, whether the video and sound must match the sound source position Sound image localization acoustic meta information indicating sound attributes, including sound image localization positions in standard production conditions and acoustic feature meta information related to the sound image localization in the case of a sound that needs to match the sound image position with the sound signal Means for generating a frame unit of
A production apparatus comprising: means for encoding and transmitting an acoustic signal under the standard production conditions together with the sound image localization acoustic meta information. A playback device that receives an acoustic signal produced according to a predetermined standard production condition, converts the acoustic signal to a new acoustic signal based on the sound image localization acoustic meta information attached to the acoustic signal, and reproduces the new acoustic signal,
Means for receiving and decoding the encoded acoustic signal and sound image localization acoustic meta information, respectively;
Based on the decoded sound image localization acoustic meta-information, means for identifying whether or not the decoded sound signal is a sound that requires matching between the video and the sound source position in units of frames;
Only when the sound needs to match the position of the image and the sound source, the sound image position suitable for the display and a predetermined number of speakers in the reception viewing environment previously associated with the standard production condition is set as the sound image localization in the standard production condition. Means for converting coordinates from position information;
Based on the acoustic feature meta-information related to the sound image localization, an acoustic signal suitable for the coordinate-converted sound image localization position is converted into an acoustic signal of a predetermined channel corresponding to a speaker in the reception viewing environment for each sound source. Means,
Means for synthesizing the converted sound signal of the predetermined channel and the sound signal of the predetermined channel that is not a sound that requires matching between the image and the sound source position, and reproducing sound from a speaker corresponding to the predetermined channel; A playback apparatus comprising:

Images