KR102058619B1 - Rendering for exception channel signal - Google Patents

Abstract

The present invention relates to a method and an apparatus for processing an object audio signal, and more particularly, to an audio signal processing method, comprising: receiving a bit string including a general channel signal and an exception channel signal; Decoding an exception channel signal and a normal channel signal from the received bit string; Generating correlation information using the decoded exception channel signal and the decoded general channel signal; Generating a gain value through at least one of a first downmix method applying the same downmix gain value using the correlation information and a second downmix method applying a variable gain value over time; An audio signal processing method may include providing the exception channel signal by outputting a plurality of channel signals using the gain value.

Description

Rendering exception channel signal {Rendering for exception channel signal}

The present invention relates to a method and apparatus for processing an object audio signal, and more particularly, to a method and apparatus for encoding and decoding an object audio signal or rendering in a three-dimensional space.

3D audio is a series of signal processing to provide a realistic sound in three-dimensional space by providing another dimension in the height direction to the sound scene (2D) on the horizontal plane provided by conventional surround audio, Commonly referred to as transmission, encoding, and reproduction techniques. Particularly, in order to provide 3D audio, a rendering technology that requires sound images to be formed at a virtual position where no speaker exists even if a larger number of speakers or a smaller number of speakers are used is widely required.

3D audio is expected to be an audio solution that is compatible with upcoming ultra-high definition televisions (UHDTVs), as well as theater sound, personal 3DTVs, tablets, smartphones, and clouds, as well as sound in vehicles that are evolving into high-quality infotainment spaces. It is expected to be applied to a variety of applications.

3D audio first needs to transmit signals of more channels than conventional ones up to 22.2 channels, which requires a suitable compression transmission technique. Conventional high quality coding such as MP3, AAC, DTS, AC3, etc. has been mainly optimized for transmitting only channels less than 5.1 channels.

In addition, in order to reproduce 22.2 channel signals, an infrastructure for a listening space equipped with 24 speaker systems is required, and technology for efficiently reproducing 22.2 channel signals in a space having a smaller number of speakers is not easy to spread in the market for a short time. On the contrary, the technology that allows the existing stereo or 5.1 channel sound source to be reproduced in a larger number of speakers, such as 10.1 channel and 22.2 channel environment, and furthermore, the sound provided by the original sound source outside of the specified speaker location and the specified listening room environment. The technology to provide a scene and the technology to enjoy 3D sound in a headphone listening environment are required. Such techniques are referred to herein as rendering, and are specifically referred to as downmix, upmix, flexible rendering, and binaural rendering, respectively.

Meanwhile, an object-based signal transmission scheme is required as an alternative for effectively transmitting such a sound scene. Depending on the sound source, it may be more advantageous to transmit on an object basis than to transmit on a channel basis. When transmitting on an object basis, the user may arbitrarily control the playback size and position of the objects. To make it possible. Accordingly, there is a need for an effective transmission method capable of compressing an object signal at a high data rate.

In addition, there may also be a sound source in which the channel-based signal and the object-based signal are mixed, thereby providing a new type of listening experience. Accordingly, there is a need for a technique for effectively transmitting channel signals and object signals together and rendering them effectively.

Finally, depending on the specificity of the channel and the speaker environment at the playback stage, exception channels may be difficult to reproduce in the conventional manner. In this case, there is a need for a technique that effectively reproduces the exception channel based on the speaker environment at the playback stage.

According to an aspect of the present invention, an audio signal processing method includes: receiving a bit string including a general channel signal and an exception channel signal; Decoding an exception channel signal and a normal channel signal from the received bit string; Generating correlation information using the decoded exception channel signal and the decoded general channel signal; Generating a gain value through at least one of a first downmix method applying the same downmix gain value using the correlation information and a second downmix method applying a variable gain value over time; An audio signal processing method may include providing the exception channel signal by outputting a plurality of channel signals using the gain value.

According to the present invention, the absence of an exceptional position or a functioning channel can be effectively reproduced according to the characteristics of the sound source. A typical example of such an exception channel is the TcP channel located directly above the head, which is a channel having a unique function of giving the effect of voice being heard directly above the head in the sky like the voice of God. This channel has a special effect unlike other cases, so if this channel is absent, it should be able to play effectively using other channels. The present invention has the effect that can be effectively compensated for even in the absence of such an exception channel. The effects of the present invention are not limited to the above-described effects, and effects that are not mentioned will be clearly understood by those skilled in the art from the present specification and the accompanying drawings.

1 is a view for explaining a viewing angle according to an image size at the same viewing distance;
2 is a layout diagram of speaker arrangement of 22.2ch as an example of a multi-channel;
3 is a conceptual diagram illustrating a process of downmixing an exception signal
4 is a flowchart of a down mixer selection unit;
5 is a conceptual diagram illustrating a simplified method in a matrix-based downmixer
6 is a conceptual diagram of a matrix-based downmixer
7 is a conceptual diagram of a path-based downmixer
8 is a conceptual diagram of a virtual channel generator

Since the embodiments described herein are intended to clearly explain the spirit of the present invention to those skilled in the art, the present invention is not limited to the embodiments described herein, and the present invention. The scope of should be construed to include modifications or variations without departing from the spirit of the invention. The terms used in the present specification and the accompanying drawings are for easily explaining the present invention, and the shapes shown in the drawings are exaggerated and displayed to help understanding of the present invention as necessary, and thus, the present invention is used herein. It is not limited by the terms and the accompanying drawings. In the present specification, when it is determined that a detailed description of a known configuration or function related to the present invention may obscure the gist of the present invention, a detailed description thereof will be omitted as necessary. In the present invention, the following terms may be interpreted based on the following criteria, and terms not described may be interpreted according to the following meanings. Coding can be interpreted as encoding or decoding in some cases, and information is a term that encompasses values, parameters, coefficients, elements, and so on. It may be interpreted otherwise, but the present invention is not limited thereto.

According to an aspect of the present invention, an audio signal processing method includes: receiving a bit string including a general channel signal and an exception channel signal; Decoding an exception channel signal and a normal channel signal from the received bit string; Generating correlation information using the decoded exception channel signal and the decoded general channel signal; Generating a gain value through at least one of a first downmix method applying the same downmix gain value using the correlation information and a second downmix method applying a variable gain value over time; An audio signal processing method may include providing the exception channel signal by outputting a plurality of channel signals using the gain value.

In addition, the first downmix method may include an audio signal processing method characterized by applying the same downmix gain value to a plurality of channels.

The first downmix method may include an audio signal processing method comprising compensating a gain value and delay information by using location information of a speaker.

In addition, the first downmix method may include an audio signal processing method characterized in that the same gain value is distributed in the equally divided space.

In addition, the second downmix method may include an audio signal processing method for estimating a moving path of a sound image on the basis of the correlation information to variably adjust a downmix gain value according to time.

Hereinafter, a method and apparatus for processing an object audio signal according to an embodiment of the present invention will be described.

1 is a view for explaining a viewing angle according to an image size (eg, UHDTV and HDTV) on the same viewing distance. Display manufacturing technology is developed, and the image size is increasing in accordance with the needs of consumers. As shown in FIG. 1, the UHDTV (7680 * 4320 pixel image 110) is about 16 times larger than the HDTV (1920 * 1080 pixel image 120). If the HDTV is installed on the living room wall and the viewer is sitting on the living room couch with a certain viewing distance, the viewing angle may be about 30 degrees. However, when the UHDTV is installed at the same viewing distance, the viewing angle reaches about 100 degrees. As such, when a large screen having a high quality and high resolution is installed, it may be desirable to provide a sound having a high sense of presence and a sense of presence suitable for the large content. In order to provide the viewer with almost the same experience as in the field, having 1-2 surround channel speakers may not be enough. Thus, a multichannel audio environment with more speakers and channel numbers may be required.

As described above, in addition to home theater environments, personal 3D TVs, smartphone TVs, 22.2 channel audio programs, automobiles, 3D video, remote presence rooms, cloud-based gaming, etc. There may be.

2 is a diagram illustrating a speaker layout of 22.2ch as an example of a multi-channel. 22.2ch may be an example of a multi-channel environment for enhancing the sound field, and the present invention is not limited to a specific number of channels or a specific speaker arrangement. Referring to FIG. 2, a total of nine channels may be provided in the highest layer 210. You can see that there are a total of nine speakers, three in the front, three in the middle and three in the surround. In the middle layer 220, five speakers in front, two in the middle position, and three speakers in the surround position may be disposed. Of the five speakers in the front, three of the center positions can be included in the TV screen. A total of three channels and two LFE channels 240 may be installed on the bottom layer 230.

As described above, in order to transmit and reproduce a multi-channel signal of up to several dozen channels, a high amount of computation may be required. In addition, a high compression ratio may be required when considering a communication environment. In addition, in a typical home, there are not many multichannel (e.g. 22.2ch) speaker environments, and many listeners have 2ch or 5.1ch setups. In the case of sending, when the multichannel is to be converted back to 2ch and 5.1ch and reproduced, not only communication inefficiency occurs but also 22.2ch of PCM signal must be stored, which may result in inefficiency in memory management.

(Requires flexible rendering)

Among the technologies required for 3D audio, flexible rendering is one of the major challenges to be able to achieve the highest quality of 3D audio. It is well known that the location of 5.1-channel speakers is very irregular depending on the structure of the living room and the layout of the furniture. Even if the speaker exists in such a non-standard position, the content creator should be able to provide the intended sound scene. In order to do this, the user must know the speaker environment in each playback environment, and the difference in the position according to the specification A rendering technique is needed to correct this. That is, the decoding of the transmitted bit string according to the decoding method does not end the role of the codec, but a series of techniques for the process of optimizing and transforming it to the user's playback environment is required.

(Flexible rendering)

With Amplitude Panning, which determines the direction information of the sound source between two speakers based on the magnitude of the signal, or VBAP (Vector-Based Amplitude Panning), which is widely used to determine the direction of sound sources using three speakers in three-dimensional space, It can be seen that flexible rendering can be implemented relatively conveniently with respect to the object signal transmitted for each object. One of the advantages of transmitting object signals instead of channels.

(Voice of God)

In a multichannel audio system, the top of center (TP) channel, the speaker above the listener's head, is often called "voice-of-god." This channel is called the voice of God because the most dramatic situation that can be obtained by using this channel is that the voice of God is heard in the sky. In addition, the effects of using this channel can vary. Examples include falling objects just above the head, firecrackers playing just above the head, or shouting at the roof of a very tall building. Or it can be said to be a very essential channel in various scenes, such as the plane disappearing from the front over the head of the viewer. In other words, by using the TpC channel, it is possible to give the user a realistic sound field that the conventional audio system cannot provide in many dramatic situations. In the case of an exception channel such as TpC, if there is no speaker at that location, compensating for it in the same way as conventional flexible rendering is not effective and hard to expect big features. Therefore, there is a need for a method of effectively reproducing a channel through a small number of output channels in the absence of such an exception channel.

When multi-channel content is played through fewer output channels, it is common practice to implement M-N downmix matrix (M is the number of input channels and N is the number of output channels). That is, when 5.1 channel content is reproduced in stereo, the downmix is implemented by a given equation. However, such a downmix implementation method generally takes a method of synthesizing by applying a downmix gain relative to speakers that are spatially close in distance. For example, TpFc of the top layer may be synthesized by downmixing to Fc (or FRc, FLc) of the middle layer. That is, by generating the virtual TpFc using these speakers Fc, FRc, and FLc, the sound corresponding to the position of the member speaker TpFc can be reproduced. However, in the case of the Tpc speaker, it is difficult to determine the direction of the front, rear, left and right with respect to the listener, so that it is difficult to determine the speaker position spatially close to the middle layer speakers. In addition, when downmixing a signal assigned to a Tpc speaker in an unstructured speaker array environment, it is sometimes effective to flexibly change the shape of the downmix matrix in conjunction with a flexible rendering technique.

As a solution to this problem, if the sound source played by the TpC is really an object of "God's voice", the object played only in the TpC or the object played around the TpC, it is preferable to downmix accordingly. However, it is also desirable to apply a specialized downmix method when a part of an object is reproduced in the entire upper layer or when the plane passes through the sky such as passing TpBR through TpC at the position of TpFL. In addition, in contrast to the above two situations, when a few limited number of speakers must be used depending on the position of the speaker, it is necessary to consider a rendering method of positioning the sound source at various angles. Elevation spectral cues exist for humans to perceive the height of a sound source.For example, depending on the height of a sound source, the appearance of the pinna is affected by the higher frequency bands and the shape of Nazis and peaks. Can be. Therefore, by artificially inserting a clue for recognizing the height of such a sound source, it is possible to effectively reproduce the sound scene of the TpC channel.

(VOG whole block diagram)

3 is a conceptual diagram illustrating a process of downmixing an exception channel signal. Exception channel signal The signal may be downmixed by analyzing a particular value of the transmitted bit string or the characteristics of the signal. An example of an exception channel signal may be a TpC signal located above the head of the listener. First, it is reasonable to apply the same downmix gain to multiple channels for stationary or ambiguous ambient signals above the head. This may downmix the TpC signal using a conventional general matrix-based downmixer 310. Secondly, in the case of the TpC signal in the mobile sound scene, the dynamic sound scene intended by the content provider becomes more static when the aforementioned matrix-based downmixer 310 is used. To prevent this, a downmix having a variable gain value may be performed by analyzing channel signals. This is called a path based downmixer 320. Finally, if the nearby speakers are not enough to achieve the desired effect, you can use spectral cues to perceive the height of a particular N speaker's output signal. This is called a virtual channel generator 330. The downmix selector 340 determines which downmix method to use by using input bit string information or analyzing input channel signals. The output signal is determined by L, M or N channel signals according to the selected downmix method.

(Downmix decision section)

4 is a flowchart of the downmix selector 340. First, the input bit string is parsed to check whether there is a mode set by the content provider. If there is a set mode, downmix is performed using the set parameter of the mode. If there is no mode set by the content provider, the speaker layout of the current user is analyzed. This is because, when the speaker layout is very irregular, as described above, if the downmix is performed only by adjusting the gain value of the neighboring channel, the content provider cannot reproduce the intended sound scene. To overcome this, people have to use various clues to recognize high altitude images.

As an embodiment of analyzing the speaker arrangement, the distance may be analyzed as a sum of distances of the position vectors of the speakers of the upper layer of FIG. Let Vi be the position vector of the i-th speaker of the upper layer of FIG. 2 and Vi 'be the position vector of the i-th speaker at the playback end. Also, if the weight is wi according to the positional importance of the speaker, the speaker position error Espk may be defined by Equation 1.

If your speaker layout is very irregular, the speaker position error Espk will be large. Thus, if the speaker position error Espk exceeds or exceeds a certain threshold, it selects the virtual channel generator. When the speaker position error is below or below a certain threshold, a matrix based downmixer or a path based downmixer is used. When the sound source to be downmixed is a channel signal, the downmix method may be selected according to the width of the estimated sound image size of the channel signal. This is because the localization blur of the person to be mentioned later is much larger than the median plane, so that a fine sound localization method is unnecessary when the parent source width is large. As an example of measuring the width of sound images of several channels, the measurement method is an example using an interaural cross correlation between two signals. However, since this requires a very complicated operation, the cross-correlation between each channel is proportional to the cross-correlation of the signal, so that the width of the sound image is relatively small using the sum of the TpC channel signal and each channel. Can be estimated. Use a matrix-based downmixer if the sum of the cross correlations between the TpC channel signal and the surrounding channel signal, C, exceeds or exceeds a certain threshold, because the width of the sound is wider than the reference; otherwise, the width of the sound is narrower than the reference. More sophisticated path-based downmixers.

(Static sound source downmixer / matrix based downmixer)

According to several psychoacoustic experiments, the phonetic position in the median plane is very different from that in the horizontal plane. A measure of the inaccuracy of the phonetic position is a localization blur, which represents the range in which the position of the sound image is not distinguished from a specific position in degrees. According to the experiments mentioned above, the voice signal has an inaccuracy corresponding to 9 degrees to 17 degrees. However, considering that the voice signal has 0.9 to 1.5 degrees in the horizontal plane, it can be seen that the sound localization in the midplane has very low accuracy. For high altitude sound images, the human-perceptible accuracy is low, so matrix downmixing is more effective than sophisticated positioning methods. Therefore, in the case of a sound image whose position does not change significantly, it is possible to effectively upmix the missing TpC channel to a plurality of channels by distributing an equal gain value to the top channels in which the speakers are symmetrically distributed.

Assuming that the top layer is the same except for the TpC channel in the configuration of FIG. 2, the channel gain value distributed to the top layer has the same value. However, it is well known that it is difficult to have a formal channel environment as shown in FIG. In a non-standard channel environment, the distribution of a certain gain value to all the aforementioned channels may cause the angle of the sound image to the position where the content is intended to be larger than the position spread value. This allows the user to recognize the wrong sound image. In order to prevent this, a process of compensating for an atypical channel environment is required.

Since the channel located in the top layer can be assumed to arrive as a plane wave at the listener's position, the conventional downmix method of setting a constant gain value can be described as reproducing the plane wave generated in the TpC channel using the surrounding channel. have. The center of gravity of the polygon with the vertices of the speakers on the plane including the top layer is the same as that of the TpC channel. Therefore, in the case of the atypical channel environment, the gain value of each channel is expressed by the equation that the center of gravity vector of the two-dimensional position vectors on the plane including the top layer of each channel to which the gain value is weighted is equal to the position vector of the TpC channel position. Can be obtained.

However, this formal approach requires a large amount of computation, and the performance difference is not large compared to the simplified method described later. The simplified method is as follows. First, N regions are divided equally around the TpC channel. An equal gain value is given to an area divided by an isometric angle, and if two or more speakers are located in the area, the sum of squares of each gain is set to be equal to the above-mentioned gain value. As an example of this, a speaker arrangement including a speaker 510 located on a plane including the top layer, a TpC channel speaker 520, and a speaker 530 located outside the plane including the top layer as shown in FIG. Suppose you have When four areas are divided into 90 degree isocenters around the TpC channel 520, each area is given a gain value such that the sum of squares is equal to one while the size is the same. In this case, since there are four areas, the gain value of each area is 0.5. If there are two or more speakers on one area, this also sets the gain value such that the sum of squares equals the gain value of the area. Therefore, the gain value of the two speaker outputs in the lower right region 540 is 0.3536. Lastly, in the case of the speaker 530 located outside the plane including the top layer, first, a gain value when projecting onto the plane including the top layer is first obtained, and the distance between the plane and the speaker is compensated using the gain and delay. do.

6 is a conceptual diagram of a matrix based downmixer 310. First, the parser 610 separates an input bit string from a mode bit provided by a content provider and a channel signal. If the mode bit is set, the speaker determiner 620 selects the speaker group. If the mode bit is not set, the speaker group having the shortest distance is the minimum using the speaker position information currently used by the user. Select. Gain and delay are compensated for the gain and delay of each speaker to compensate for the difference between the speaker group set in the compensator 630 and the speaker arrangement of the actual user. Finally, the downmix matrix generator 640 applies the gain and delay output from the gain and delay compensation unit 630 to downmix the channel output from the parser to other channels.

(Dynamic sound source downmixer / path based downmixer)

7 is a conceptual diagram of a dynamic sound source down mixer 320. First, the parser 710 parses the input bit string and transmits the exception channel signal and the plurality of channel signals in the vicinity to the path estimator 720. In the case of the plurality of channel signals, the path estimator 720 estimates a correlation between channels and estimates a change in channels having a high correlation as a path. The speaker selector 730 selects speakers having a predetermined distance or less from the path estimated using the path estimated by the path estimator 720. The location information of the selected speakers is transmitted to the downmixer 740 and downmixed according to the speaker. As an example of the downmix method, vector base amplitude panning (VBAP) is an example.

(Virtual channel generator)

8 is a conceptual diagram of a virtual channel generator 330. The input bit string is parsed into an exception channel signal through a parser 810. The parameter extractor 820 extracts a parameter using a generalized head transfer function or a provided personalized head transfer function embedded in the transferred exception channel signal. As an example of the parameter, the frequency and magnitude information of the Nazi or peak of a specific spectrum or the amount of a specific frequency may be a level difference and the amount may be a phase difference. The virtual channel based downmixer 830 performs downmixing based on the transmitted parameters. Examples of such a downmix include filtering the head transfer function or complex panning, which performs panning by dividing into a specific band at the entire frequency.

The audio signal processing method according to the present invention can be stored in a computer-readable recording medium which is produced as a program for execution in a computer, and multimedia data having a data structure according to the present invention can also be stored in a computer-readable recording medium. Can be stored. The computer readable recording medium includes all kinds of storage devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and may also be implemented in the form of a carrier wave (for example, transmission over the Internet). Include. In addition, the bitstream generated by the encoding method may be stored in a computer-readable recording medium or transmitted using a wired / wireless communication network.

As described above, although the present invention has been described by way of limited embodiments and drawings, the present invention is not limited thereto and is intended by those skilled in the art to which the present invention pertains. Of course, various modifications and variations are possible within the scope of equivalents of the claims to be described.

210: upper layer
220: middle layer
230: floor layer
240: LFE channel
310: matrix based downmixer
320: path based downmixer
330: virtual channel generator
340: down mixer selector
510: Speaker located on the plane containing the top layer
520: TpC Channel
530: Speaker located outside the plane containing the top layer
610: parser
620: speaker determination unit
630: gain and delay compensation unit
640: downmix matrix generator
710: parser
720: path estimation unit
730 speaker selection unit
740: downmixer
810: Parser
820: parameter extraction unit
830 virtual channel based downmixer

Claims (6)

As an audio signal processing method,
Receiving a bit string including a normal channel signal and an exception channel signal;
Decoding an exception channel signal and a normal channel signal from the received bit string;
Obtaining first information relating to respective positions of a plurality of speakers from the received bit strings;
Acquiring second information about a position at each reproduction end of the plurality of speakers;
Deriving a gain value based on the first information and the second information; And
Outputting the exception channel signal using the gain value;
Deriving the gain value,
Compensating for the gain and delay based on the second information. delete The method of claim 1,
Each of the first information and the second information,
The audio signal processing method comprising information on each azimuth or altitude of the plurality of speakers. The method of claim 1,
Deriving the gain value,
And inducing a sum of an absolute value of a difference between a position of the speaker according to the first information and a position of the speaker according to the second information. The method of claim 1,
Deriving the gain value,
Computing a gain value for the speaker of the playback stage using VBAP (Vector Based Amplitude Panning). The method of claim 1,
The exception channel signal is,
An audio signal processing method comprising a VoG (Voice of God) signal.

Images