JP2004526355A - Audio channel conversion method - Google Patents

Abstract

A process for translating M audio input channels representing a soundfield to N audio output channels representing the same soundfield, wherein each channel is a single audio stream representing audio arriving from a direction, M and N are positive whole integers, and M is at least 2, generates one or more sets of output channels, each set having one or more output channels. Each set is associated with two or more spatially adjacent input channels and each output channel in a set is generated by a process that includes determining a measure of the correlation of the two or more input channels and the level interrelationships of the two or more input channels.

Description

ＸとＹは相関関係がないから
XY = 0
従って、
【数２】

これは、ＸとＹは相関関係がなく、主要チャンネルＡのトータルエネルギーは信号ＸとＹのエネルギーの和だからである。
【００５４】
同様に、
【数３】

Ｘ，Ｙ及びＺは互いに相関関係はないので、ＡとＢの外積は、
【数４】

【００５５】
従って、独立で相関関係のない信号を含むことのある２つの隣接する主要チャンネルで同等に共有された出力信号の場合について言うと、信号の外積値を平均したものは、各チャンネルにおける共通信号成分のエネルギーに等しい。もし、共通信号が等しく共有されていない場合は、すなわち、主要チャンネルの一方に偏っていた場合は、外積値を平均したものは、Ａ及びＢにおける共通成分のエネルギーの幾何平均となる。Ａ及びＢから、そのチャンネルの増幅係数を２乗平均により正規化することで、各チャンネルにおける共通成分の推定値を得る。実際の時間平均は、継続的に状態を反映させるために、最適な減衰時定数を持つ積分器（leaky integrator)により計算される。時定数による平滑化は、非線形化や減衰時間にオプションを持たせることにより、さらに精密に行うことができる。また、マルチバンドシステムにおいては、周波数により計量しても良い。
【００５６】
共通エネルギーの高次数での計算
３以上の入力をもつデコーディングモジュールの共通エネルギーを導き出すには、すべての入力信号の外積値の平均値を計算することが必要である。単純に入力のペア毎に計算を進めたのでは、入力の各ペアと全体の共通信号との間での別々の出力信号を区別することができなくなる。
【００５７】
例えば、互いに相関のない信号Ｗ，Ｙ，Ｚと、共通する信号Ｘからなる３個の主要チャンネル、Ａ，Ｂ及びＣを考えてみる。
【００５８】
A = X + W
B = X + Y
C = X + Z
平均外積値を計算すると、Ｗ，Ｙ及びＺの組合せとなる項はキャンセルされ、２次の計算により、Ｘ^３の平均値が残る。
【００５９】
【数５】

【００６０】
しかしながら、期待した通り、もしＸの時間平均がゼロならば、その３乗平均もゼロになる。ゼロ以外のＸの値の場合はすべて正となるＸ^２を平均する場合とは異なり、Ｘ^３はＸと同じ符号を持つので、正と負の寄与分はキャンセルされてしまう。モジュールの入力数が奇数の場合に対応して、Ｘが奇数乗となる場合はすべてこのことが当てはまるのは、明らかである。しかし指数が２以上の場合でも、間違った結果を導き出す。例えば、成分が（Ｘ，Ｘ，−Ｘ，−Ｘ）となる４入力の場合は、平均外積値は（Ｘ，Ｘ，Ｘ，Ｘ）の場合と同じとなる。
【００６１】
この問題は、積の平均値を算出する手法に変更を加えることにより解決されている。平均化する前に、積の絶対値を採用することにより各積の符号を捨て去る。積の各項の符号は調べられる。もし、すべての符号が同じであれば、その積の絶対値を平均演算に用いる。もし、いずれかの符号が他と違っていた場合、積の絶対値の負の値を平均する。同じ符合の組みあわせが起こり得る数と、違う符合が起こり得る数とが違っているので、その補正のため、同じ符号の組み合わせの数に対する違う符合の組合せの数の比からなる重み係争を、積の絶対値を無効にするため適用する。例えば、３入力のモジュールは同じ符号となる組合せが、８通りの組合せから、６通りの違う符合の組み合わせを引くと、２通りあり、その結果、スケールファクターは２／６＝１／３となる。この補正は、デコーディングモジュールのすべての入力に共通入力信号成分がある場合にのみ、積算または加算された積を正の方向に増大させる結果となる。
【００６２】
しかしながら、違った系列のモジュールの平均値を比較可能なものとするため、それらはすべて、同じ次数を持たなければならない。従来の２次の相関は２入力の積つまりエネルギーまたはパワーの次元を持つ量を生じる。従って、より高次の相関において平均化する項目は、パワーの次元を持つよう修正されなければならない。従って、ｋｔｈの次数の相関に対しては、平均化する前に各積の絶対値は２／ｋのパワーに次数を上げておかなくてはならない。
【００６３】
もちろん、次数を考えないで、必要ならモジュールの各入力ノードのエネルギーを対応するノードの信号の２乗平均として計算することもできる。そうすれば、最初にｋｔｈパワーに次数を上げ、続いて２次の値に下げる必要はない。
【００６４】
共有ノード(Shared Nodes)：隣接レベル(Neighbor levels)
２乗平均値及び主要チャンネル信号の修正された外積値を用いることにより、共通出力チャンネルの信号エネルギーの量は計算することができる。上記例では一次補間処理を必要とするが、もし、Ａ／Ｂ（／Ｃ）の内の１つ以上のノードは、共通の信号成分を持ち、それが他のどの信号とも相関関係を持たないような他のモジュールと共通していたとすると、計算は本質的にどんなイメージ伸張効果も受けないため、上述の外積の平均値は影響を受けないであろう（注記：２出力信号が、非相関でない場合は、これらはいくらかのデコーダーを伸張する傾向をもつ。しかし人間の耳にも同じような効果があるので、それでもシステムの動作は人間の聴覚に忠実でありつづける。）。
【００６５】
いったん、各デコーディングモジュールが各々その主要ノードで共通出力チャンネル信号のエネルギーの推定値を算出してしまうと、監視機能により、隣接するモジュールにお互いの共通エネルギーを伝達する。そしてその共通エネルギーのポイントで、下記に記すような過程で出力チャンネルの信号の抽出を行うことができる。ノードでモジュールにより用いられる共通エネルギーの計算は、異なる次数を持ったモジュールと重複する可能性を持つ階層構造を考慮しておき、同じノードを共有するあらゆる低次数のモジュールの共通エネルギーの算定値から、高次数のモジュールの共通エネルギーを減算する必要がある。
【００６６】
例えば、信号エネルギーとしてＸ^２を持ち、２つの隣り合う主要チャンネルで２つの水平方向を代表するＡとＢとに加えて、垂直方向を代表する主要チャンネルＣと、さらに、内部に向かい（すなわち、Ａ，Ｂ及びＣの制限内に置かれ）、中間的あるいは派生的な出力チャンネルの存在を考える。３個の入力モジュールの共通エネルギーは、入力（Ａ，Ｂ，Ｃ）と共に、Ｘ^２となる。しかし、２個の入力モジュール（Ａ，Ｂ），（Ｂ，Ｃ）及び（Ａ，Ｃ）も同じとなる。もしＡに関連するモジュール（Ａ，Ｂ，Ｃ），（Ａ，Ｂ）及び（Ａ，Ｃ）を単純に加えると、Ｘ^２とはならないで３Ｘ^２という結果になる。共通ノードのエネルギーの計算が正しくなるように、より高い次数のモジュールは、重複する低次数のモジュールの共通エネルギーの算定値から、最初に減算される。したがって、高次数のモジュール（Ａ，Ｂ，Ｃ）の共通エネルギーＸ^２は、２個の入力モジュールにおける共通エネルギーの算定値から減算され、その結果各々０となり、ノードＡにおける共通エネルギーはＸ^２＋０＋０＝Ｘ^２と算定される。
【００６７】
出力チャンネル信号の抽出
すでに注記したように、伝達されてきたチャンネルから線形的な方法で出力チャンネルの総体を回復させる過程は、基本的には出力チャンネルを導き出すための重み付けされた主要チャンネルの総計を成すマトリックスを形成することである。マトリックスにおけるスケールファクターの最適の選択は、一般的には信号に依存する。実際に、もし現在動作中の出力チャンネルの数が伝達されてきたチャンネル（ただし、違う方向を表している）の数と同じであれば、システムは正確に決まってしまい、数学的に効果的なマトリックスを解読することが可能となり、信号源とは独立した信号を再生する。動作中の出力チャンネルの数が、主要チャンネルの数より大きいときであっても、擬似的に逆変換するマトリックスを計算することは可能である。
【００６８】
残念ながら、この取り組み方法には問題がある。この取り組み方法において、特にマルチバンドをベースとする場合で、高い精度の浮動小数点を実現しようとする場合の計算をする上での負荷は決して些細な問題ではない。中間信号は最も近傍にある主要チャンネルに向けられると想定されているにもかかわらず、数学的な逆変換すなわち効果的なマトリックスの擬似的な読解は、一般的には、ノード共有効果のため、すべての主要チャンネルから各出力チャンネルへの寄与を伴う。もしデコーディングに不完全さがあるならば、そしてそれは実際には避けられないことではあるが、空間的に遠くに隔てられた出力チャンネルから主要なチャンネルが再現される可能性があり、それは決して望ましいことではない。加えて、擬似的な逆変換の計算は、最低２乗平均エネルギーの解(minimam-RMS-energy solutions)を導く傾向にあり、これは、分離を最低限に押さえて、音声を最大限に拡散させるもので、本来の意図に反するものである。
【００６９】
そこで、実用的で、空間的なデコーディング誤差が本質的に内在することを認容するデコーダを実現するため、信号を検出するために用いたのと同じモジュール構成を信号再生に採用する。
【００７０】
以下は、出力信号がデコーディングモジュールにより再現される再生手順の詳細である。ここで、モジュールにつながる各出力チャンネルの効果的な位置は、信号を物理的な位置に向けさせること、すなわち、その方向に関連する効果的なマトリックスエンコード係数の比の算出のために必要とする増幅係数、により表示されると仮定されることに注意しなければならない。ゼロで除算される問題を避けるため、係数は、入力チャンネルのマトリックス係数（普通は１）のすべての２乗平均（ＲＭＳ）に対する１つのチャンネルのマトリックス係数の比として計算される。例えば、ＬとＲの入力を持つ２入力モジュールにおいて、用いられるエネルギー比は、ＬとＲのエネルギーに対するＬのエネルギー("L-ratio")であり、これは０から１の間に収まる行儀の良いものである。もし２入力デコーディングモジュールが、（１．０，０），（０．８９，０．４５），（０．７１，０．７１），（０．４５，０．８９）及び（０，１．０）のエンコーディングマトリックスのペアからなる５個の出力チャンネルを持っていたとすると、各スケールファクターのペアが２乗平均値１．０をとるから、対応するL-ratiosは、１．０，０．８９，０．７１，０．４５及び０となる。
【００７１】
デコーディングモジュールの各入力ノード（主要チャンネル）の信号エネルギーから、隣接するデコーディングモジュールからの要求により、ノードを共有するすべての信号エネルギーが減算され、その結果が、残った計算に使うための正規化された入力信号レベルとなる。
【００７２】
主要(dominant)な方向の指標は主要方向のベクトル和として計算され、相対的なエネルギーで重み付けされる。２入力モジュールに対しては、これは、正規化された入力信号レベルであるL-ratioに単純化される。
【００７３】
主要方向(dominant direction)を区分する出力チャンネルは、ステップ２の主要方向のL-ratioと比較することで決定される。例えば、上記５出力デコーディングモジュールの入力のL-ratioが０．７５であれば、０．８９＞０．７５＞０．７１なので、２番目と３番目の出力チャンネルが主要な信号方向となる。
【００７４】
主要信号(dominant signal)を直近の区分されたチャンネルと関連付けるパンニングスケールファクターはチャンネルの反主要信号(anti-dominant signal)レベルとの比から計算される。特別な出力チャンネルと結びついた、この反主要信号(anti-dominant signal)は、対応するデコーディングモジュールの入力信号が出力チャンネルの反主要マトリックス(anti-dominant matrix)のスケールファクターと共に行列に組み込まれたときに答えとして出てくる信号である。出力チャンネルからの反主要マトリックス(anti-dominant matrix)のスケールファクターは、１つの主要信号(dominant signal)が該当する出力チャンネルへ送られるときゼロ出力となるような２乗平均和(RMS sum)＝１．０を含むスケールファクターである。ここで、出力チャンネルのエンコードマトリックスのスケールファクターが（Ａ，Ｂ）であれば、このチャンネルの反主要スケールファクター(anti-dominant scale factors)はちょうど（Ｂ，−Ａ）となる。
【００７５】
証明
もし、１つの主要信号(dominant signal)が、エンコードスケールファクター（Ａ，Ｂ）を持つ出力チャンネルに送られたとすると、信号は（ｋＡ，ｋＢ）に増幅される。ここで、ｋは全体としての信号増幅係数である。従って、このチャンネルの反主要信号(anti-dominant signal)は（ｋＡ＊Ｂ−ｋＢ＊Ａ）＝０となる。
【００７６】
従って、もし、主要信号(dominant signal)が、２乗平均(RMS)＝１となるよう正規化された入力増幅係数をもつ２入力のモジュールの入力信号（x(t),y(t))からなるとすると、展開された主要信号(dominant signal)はdom(t)=Xx(t)+Yy(t)となる。もし、信号のポジションが、各々（Ａ，Ｂ）と（Ｃ，Ｄ）のマトリックススケールファクターを持つ出力チャンネルとして区分された場合は、主要信号(dominant signal)のスケールファクターの前者のチャンネルに対するスケーリングdom(t)は：
SF(A,B)=sqr((DX-CY)/((DX-CY)+(BX-AY)))
一方、後者のチャンネルに対応する主要信号(dominant signal)のスケールファクターは：
SF(C,D)=sqr((BX-AY)/((DX-CY)+(BX-AY)))
主要方向(dominant direction)は１つの出力チャンネルから他の出力チャンネルへ送られるので、これらの２つのスケールファクターはパワーの合計を一定に保ちながらゼロから１までの間で互いに反対方向に動く。
【００７７】
反主要信号(anti-dominant signal)が算出され、適切なゲインスケーリング(gain scaling)と共に、非主要チャンネル(non-dominant channels)に送られる。反主要信号(anti-dominant signal)は、あらゆる主要信号(dominant signal)が除かれたマトリックスの信号である。もし、デコーディングモジュールへの入力が、正規化された振幅（X,Y)を持つ（x(t),y(t))であるとすると、非主要出力チャンネル(non-dominant output channels)の位置とは無関係に、主要信号(dominant signal)はdom(t)=Xx(t)+Yy(t)となり、反主要信号(anti-dominant signal)はdom(t)=Yx(t)-Xy(t)となる。
【００７８】
主要信号／反主要信号(dominant/anti-dominant signal)の分配に加えて、基本的にはすでに議論した出力マトリックスのスケールファクターであって、パワーを保持するために倍率を定めた受動(passive)マトリックスを用いて、２番目の信号分配が計算される。
【００７９】
デコーディングモジュールの入力信号相互間の相関は入力信号の外積値を平均したものを入力レベルを正規化してものの平方根で除算することで算出される。
【００８０】
展開処理の過程に戻ると、最終的な出力信号は、クロスフェードファクターを導くためのデコーディングモジュールの入力信号の相関係数を使って、主要受動信号(dominant passive signal)の分配での重み付けされたクロスフェードの和として算出される。相関係数＝１のときは、主要信号／反主要信号(dominant/anti-dominant signal)の分配が排他的に用いられる。相関係数が小さくなるにつれて、出力信号の一群はクロスフェードにより受動的な配置にまで広げられて、小さな正の相関関係を持つにいたる。この値は、デコーディングモジュールに接続された出力チャンネルの数に依存するが、０．２から０．４の間の値を典型とする。さらに相関係数がゼロに向かって下がっていくと、受動的な振幅の出力配分はだんだん減少し、そのような信号に対する人間の耳を模擬しながら、出力チャンネルのレベルを下げる。
【００８１】
垂直方向の処理
これまで述べてきた処理のほとんどは、出力と主要チャンネルの方向とを考慮しないで、近接する主要チャンネルから出力チャンネルを展開するのに適用するものであった。しかしながら、耳は水平に配置しているので、チャンネル間での相関について、水平方向より垂直方向に対してより鈍感になっている。耳における機能の忠実性を維持するために、最初にひずみ機能(warp function)を適用する相関信号の処理のような、垂直方向の入力に用いる処理装置の、補間における制約の相関関係を緩和させることが望ましいであろう。しかしながら、このことは、水平チャンネルに対する処理と同じ処理を使うことが、聴覚に対して悪影響を与えることではない。同じ処理を使うことは、全体としてデコーダの構成を単純にするであろう。
【００８２】
厳密に言うと、垂直方向の情報は上部からと下部からの両方の音声が含まれる。そして、デコーダの構成はどちらにも同様に作用する。しかし、実際には、普通は、ほんの少しの自然な音声が下部からくるだけである。従って、このような下部からの信号に対する処理やチャンネルは、システムの空間的な認識の忠実度を損なうことなく省略できるであろう。
【００８３】
この概念は、従来からある５．１チャンネルのサラウンドに対するチャンネル変換の適用に対して、もちろん、垂直チャンネルが欠けているが、実用的な意義を持っている。しかしながら、これは、多数またはすべての水平チャンネルを横切って出力するフライオーバー効果のように、垂直方向の情報を含むこともできる。従って、隣り合わないチャンネルやチャンネルのグループの相関を探すことで、このような信号源から仮想的な垂直チャンネルを展開することは可能である。このような相関関係の存在下で、これらは、リスナーの下からより、上部からの垂直方向の情報の存在を表現している。ある場合は、恐らく対象とするリスナーの環境に調和させることで、残響発生装置から仮想的な垂直方向の情報を導き出すことが可能となる。一度５．１チャンネル源から仮想的な垂直チャンネルを展開しあるいは導き出せば、先に説明した２４チャンネル配置のような大きな数のチャンネルに対しても、あたかも垂直チャンネルが供給されているかのように、展開が可能となる。
【００８４】
方向性メモリー
以上に述べたデコーディングモジュールの制御はPro Logicデコーダーのような能動的２：Ｎデコーダーに似ているという１つの観点がある。それは、その処理においてメモリーだけが、基本的なコントロール信号を導き出す平滑化されたネットワーク内にある、という観点である。いずれかの時点で、ただ１つの主要方向(dominant direction)と入力相関の値があり、これらの値によって信号の展開処理がなされるというものである。
【００８５】
しかしながら、（典型的なカクテルパーティーのような）特に複雑な音響環境において、人間の耳は、一定の位置記憶または位置に対する慣れがあることが明らかになっており、そのため、明らかに特定された位置から与えらた簡潔な主要音が違ったもの、つまり、同じ音源から来るときに認識される概略の方向と比べて、はっきり位置が特定できないようなものとして認識される。
【００８６】
デコーディングモジュールに（現にPro Logicデコーディングにおいても）この効果を模擬することは、最新の主要方向のトラックを保持する明白な機構を付け加え、方向的に紛らわしい信号の状況の合間に、最新の主要方向に向けて出力信号の配分に重み付けすることで、可能である。このことで、音の再現の認識において、複雑な信号群の分離性と安定性を強めることができる。
【００８７】
信号相関の修正とチャンネルミキシングの選択
説明した通り、各デコーディングモジュールの拡散の決定は、入力信号の相互相関を基礎としている。これでは、ある条件の下で、出力信号の内容の量を小さく見積もってしまうことがある。このことは、例えば、到達時間が少し異なり、中央に向かわず同じ振幅でない、自然な信号の場合に相関値を下げてしまう、という結果になるというような形で起こる。この効果は、チャンネル間の遅れを相応に引き伸ばし、間隔を広く開けてマイクロフォンを用いた場合に、より強調される。これを補償するために、処理のための要求MIPSを少し高めるという負担を生じるが、相関計算をチャンネル相互の時間遅れのレンジをカバーするよう拡張してもよい。さらに、聴覚神経における神経細胞は約１msecの実効的な時定数を持っているので、１msecの時定数の平滑回路で整形した音声の一次的な平滑作用により、もっと現実的な相関値を得てもよい。
【００８８】
さらに、コンテントのプロデューサーが、ほとんど相関のないチャンネルと共に既存の５．１チャンネルプログラムを持っている場合、チャンネル変換デコーダにより処理されるとき拡散の等価性は、隣り合うチャンネルを少し混合することにより向上し、そのことで、相関が上がり、チャンネル変換デコーディングモジュールが、中間出力チャンネル間でより平等に拡散するようになる。このような混合は、例えば、ダイアログトラックをコンパクトに維持するために中央正面のチャンネルの信号は混合しないでおくというように、選択的に行うことができる。
【００８９】
ラウドネスの圧縮と拡張
エンコーディング処理は、多数のチャンネルを混ぜて、少数のチャンネルにすることを伴う、これは、何らかの形でゲイン補償がない場合は、エンコードされた信号を省略してしまう可能性を示している。この問題は、従来のマトリックスエンコーディングにおいても同様に存在していた。しかし、混合され出力チャンネルに与えられる数がより大きいため、チャンネル変換の場合は潜在的により大きな問題となる。このような場合において、信号の省略をなくすため、全般的なゲインスケールファクターをエンコーダから導き出し、デコーダのコード化されたビットストリームに移送する。普通は、この値は０dBであるが、信号の省略を避けるため、デコーダがゲイン補正と等価な量を供給し、エンコーダによりゼロでない減衰値が与えられる場合がある。
【００９０】
もし、デコーダが、このようなスケールファクタープログラムが無い既存の多チャンネル（例えば、既存の５．１チャンネルサウンドトラック）にこの処理を行うために用いられるとするならば、想定される値（０dBと推定）を持つ固定スケールファクターをオプション的に用いても良く、または、シグナルレベル及び／又はダイナミクスをベースとした拡張機能を適用しても良い。あるいは、ダイアローグを正規化する値のような、有効なメタデータをデコーダゲインの調整のために使用することも可能であろう。
【００９１】
本発明及びその種々の形態は、アナログ回路として、あるいは、さらに高い可能性で、ディジタルシグナルプロセッサ、プログラムを組み込んだ汎用ディジタルコンピュータ及び／又は特殊用途のディジタルコンピュータにより形成されたソフトウエアファンクションとして実現される。アナログ信号とディジタル信号との信号の流れのインターフェースは、それに適したハードウェア及び／又はソフトウエアにおける機能及び／又はファームウエアで実行される。
【図面の簡単な説明】
【００９２】
【図１】図１は、デコーダの理想化された配置を示す概略平面図である。【Technical field】
[0001]
The present invention relates to a method for processing an acoustic signal. Specifically, it converts M audio input channels representing an acoustic space into N audio output channels representing the same acoustic space, and each channel is a single audio channel representing a sound from one direction. The acoustic flow, M and N are positive integers, and M is at least 2 or more.
[0002]
Technical background
Although humans have only two ears, we hear sound as a three-dimensional being, with some localization cues, such as head related transfer functions (HRTFs) and head movements. Therefore, reproducing a completely faithful sound requires reproducing and maintaining a complete three-dimensional acoustic space, or at least requiring perceptual cues therefor. Unfortunately, acoustic recording techniques are not oriented in the direction of capturing a three-dimensional acoustic space, in the direction of capturing a two-dimensional acoustic plane, or even in the direction of capturing a one-dimensional acoustic line. Current acoustic recording technologies aim to precisely capture, store and represent zero-dimensional discrete acoustic channels.
[0003]
Since Edison's first phonograph invention, much of the effort to increase fidelity has focused on improving the imperfections of grooved cylindrical / disk media he created for analog modulation. I was These imperfections included limited and non-uniform frequency response, noise, wow and flutter, rotational accuracy, abrasion, dust, and loss in copying. Traditional individual channel quality despite some fragmentary attempts at individual improvements, including electronic amplifiers, tape recorders, noise reduction, and record players more expensive than a few cars It was clear that the technical problems could not be finally solved until there was a remarkable development in digital recording in general, and in particular until the introduction of audio compact discs. Since then, apart from some efforts to further improve the quality of digital recordings to 24 Bit / 96 kHz sampling, the main approach in researching sound reproduction has been to use primarily perceptual coders. This focuses on reducing the amount of data required to maintain the quality of each channel and increasing spatial fidelity. The latter problem is the issue (to be solved) in this document.
[0004]
Efforts to increase spatial fidelity have been pursued in two directions: attempts to convey perceptual cues in the entire space and attempts to convey information that actually approximates the original acoustic space. ing. Examples of systems that employ the former approach include virtual environment systems based on binaural recordings and two speakers. Unfortunately, such systems exhibit some imperfections. In particular, it is required that the sound be localized in a certain direction and that headphones be used, or that the position of the listener be stationary and alone.
[0005]
A feasible option for providing spatial sound to multiple listeners, whether in a living room or a venue used for commercial purposes such as a cinema, actually approximates the original acoustic space there It was an attempt. Given the essential recording characteristics of individually independent audio channels, it is important to note that many older approaches have involved a method called conservative, which increases the number of audio representation channels. Not surprising. Speech expression systems include three-speaker soundtracks from the early 50's, traditional stereo sound, quadraphonic systems from the 60's, 5-channel discrete magnetic soundtrack on 70mm film, and AC-3 5.1-channel sound from the 90's. And recent, Surround-EX6.1 channel sounds. "Dolby", "Pro Logic" and "Surround EX" are trademarks of Dolby Laboratories Licensing Corporation. To some extent, these systems have enhanced spatial reproducibility over single source sound representations. However, mixing a large number of audio channels creates a time and cost penalty for the producer, generally resulting in perceived as one of the discrete and scattered audio channels rather than an acoustic space as a continuum. Was. Dolby Pro Logic decoding features are described in US Patent 4,799,260, which is incorporated by reference herein in its entirety. Details of AC-3 are published in "Digital Audio Compression Standard (AC-3)," Advanced Television System Committee (ATSC), Document A / 52, December 20, 1995. (Available on the Internet World Wide Web at www.atsc.org/Standards/A52/a_52.doc). Also see the Errata sheet of July 22, 1999 (available on the World Wide Web at www.dolby.com/tech/ATSC_err.pdf)
[0006]
Features of the present invention
The basis for reproducing arbitrary distributions in a sourceless medium is given by Gauss's theorem, which guarantees that the distribution of pressure along the boundary of the region and the wave field within the boundary is completely determined . This conceptually means placing a living room and a sound-impermeable wall in a concert hall, placing an infinite number of infinitely small microphones outside the wall, connecting them to amplifiers, It suggests that it is possible to reproduce the acoustic space of a concert hall in the area of a living room by electronically expressing a sound-transparent wall by connecting it to the corresponding loudspeaker just inside. I have. By intervening a suitable recording medium between the microphone and the speaker, a complete but rigorous, but impractical, 3D sound reproduction system is realized. The remaining design challenge is to make this system practical.
[0007]
The first approach to practical use noted that the bandwidth of the signal of interest was limited to about 20 kHz, and that by accepting improvements in the spatial and temporal sampling theorems, it was possible. Become. The latter means that no information is lost if a temporally continuous wave whose band is limited is sampled discretely at least twice as long as the original wave. The former theorem requires similar considerations that spatial sampling intervals must be at least twice as dense as the minimum wavelength to avoid loss of information. The wavelength of a 20 kHz sound wave in air is about 3/8 inch, suggesting that a rigorous 3D sound system would require microphones and loudspeakers to be spaced no more than 3/16 inch apart. . When applied to a typical 9 foot x 12 foot room surface, this results in 2.5 million channels, a significant improvement over an infinite number, but still impractical at this time. It should be noted that a basic approach using a discrete channel arrangement as spatial sampling has been established, and an acoustic space is reproduced from this by using appropriate interpolation.
[0008]
When the characteristics of the acoustic space are clarified, it is possible in principle to derive an optimum output for any output speaker for the decoder. The channels feeding such a decoder are referred to variously herein as "main" channels, "transmission" channels, or "input" channels. And any output channel at a location that does not correspond to one location of the primary channel is referred to as an "intermediate" channel. The output channel may be arranged in accordance with the position of the main input channel.
[0009]
Therefore, it is desirable to reduce the number of discrete or dominant channels of spatial sampling. One basis for this is the fact that above 1500 Hz, the ear cannot follow each cycle, but follows the envelope component. This reduces the total number of channels in a 9 foot x 12 foot room to about 6000 channels. It is effective to reduce about 2.49 million channels compared to the first arrangement.
[0010]
In any case, it is theoretically possible to further reduce the spatial number of channels to be sampled by appealing to psychoacoustics and by limiting it to limited locations. The horizontal resolution is an arc of about 1 degree for a sound going to the center. The corresponding vertical resolution is about 5 degrees. If this density were properly applied to the sphere, the result would still be hundreds to thousands of channels.
[0011]
Disclosure of the invention
The method according to the invention converts M audio input channels representing an acoustic space into N audio output channels representing the same acoustic space, each channel representing sound from a certain direction. One acoustic stream, M and N are positive integers, and M is at least two or more. And generating one or more sets of output channels, each set having one or more output channels, each set being associated with two or more spatially adjacent input channels, and each output in the set being Channels are created by a process that includes means for correlating two or more input channels, and determining a level of interaction of the two or more input channels.
[0012]
One feature of the present invention is that a plurality of sets of output channels are associated with two or more input channels, and with the method according to the present invention each set of one or more output channels is associated with a number of associated input channels. According to a hierarchical order as ranked accordingly, each set of output channels determines the correlation of the associated input channels. Here, the higher the number of input channels, the higher the ranking, and the processing method is set according to a hierarchical order. Further, the method according to the present invention is characterized in that the processing is performed in consideration of the result of the processing procedure of the higher-ranked set.
[0013]
In the playback and recording aspects of the present invention, each of the M acoustic input channels indicating arrival from one direction is represented by a passive matrix that encodes the nearby amplitude-panned in each sound source direction (ie, , It is assumed that the direction of the sound source is mainly related to the nearest main channel), without requiring the addition of side chain information (side chain information or auxiliary The use of information is optional) and is assumed to be compatible with existing mixing technologies, consoles and formats. However, such sound sources are encoded by a passive matrix (which thus constitutes an "effective encoding matrix"), which is a common recording technique that inherently generates such sound sources. It could be generated using In the playback and recording aspects of the present invention, possible time delays are tolerated, and sound coming from intermediate directions essentially communicates information to nearby microphones (for a group of microphones placed horizontally, Or, specifically, to communicate information to a nearby pair of microphones) or to a sound source normally recorded by five directional microphones.
[0014]
The decoder or the decoding process in the present invention is executed by a pair of modules or module functions (hereinafter, referred to as “decoding module”) that are processed in a grid. Each of the decoding modules is used to generate one or more output channels (or one or more output channels) from two or more primary channels that are spatially closest and related to the decoding module. To generate a control signal that can be The output channel represents the proportion of the audio signal that is spatially closest to the primary channel associated with the particular decoding module. As described in detail below, the decoding modules are loosely coupled to each other in the sense that they share nodes and that there is a hierarchy. The modules are hierarchically ordered according to the number of primary channels involved (the module with the highest number of primary channels involved is ranked highest). The supervisory function governs the modules so that common node signals are fairly distributed and higher order decoder modules affect the output of lower modules.
[0015]
Each decoder module may be configured to produce an output signal directly, or for each decoder module to generate variable matrix coefficients, or all output signals, together with control signals generated by other decoder modules. A matrix may be included that generates a control signal used to change the scale factor of the input or output to the fixed matrix.
[0016]
The decoder module simulates the work of the human ear and attempts to provide perceptually explicit repeatability. Each decoder module may be implemented in either a wideband or multiband configuration or function. In the latter case, either a continuous filter bank or a block structure may be used, for example, using a transform-based processor, or using a process having the same essential elements in each band.
[0017]
The basics of the invention relate in principle to the spatial transformation of M input channels into N output channels, where M and N are positive integers and M is at least 2 or more. However, another feature of the present invention is that the number of speakers received from the N output channels is carefully dependent on a virtual image that makes the acoustic image feel in a space other than where the loudspeakers are located. By doing so, it can be reduced to a realistic quantity. The most common use of a virtual image is to pan a monaural signal between the channels of two speakers, thereby reproducing an acoustic image in stereo between the two speakers. The virtual image approach is considered a viable technology for performing against a large number of people with a small number of channels, as the listener must be at or near the same distance from the two speakers. Not. In cinemas, for example, the left and right front speakers are too far apart to give many audiences the effective illusion of being centered sound. Therefore, since the center channel is important as a conversation sound source, a physically centered speaker is used instead.
[0018]
However, as the density of loudspeakers increases, many audiences can reach the point where virtual images can be created between any pair of loudspeakers, at least to the point where the sound image can be moved smoothly. Will do. With enough loudspeakers, gaps between loudspeakers are less noticeable. Such an array of speakers has a capability almost indistinguishable from the 2 million array derived above.
[0019]
To test the features of the present invention, we placed five speakers on each wall in a horizontal orientation, with a common corner speaker for a total of 16 speakers. In addition, 6 speakers are placed in a ring at a vertical angle of 45 degrees above the head of the audience, and one speaker is placed directly above, making a total of 23 speakers, and adding a subwoofer / LFE channel to 24 speakers. , And signals are sent from a PC set for 24 channel reproduction. According to current industry terminology, this system may be a 23.1 channel system, but will be briefly referred to herein as a 24 channel system.
[0020]
FIG. 1 is a schematic plan view showing an idealized arrangement of decoders as described above as a test arrangement. The five wide-range horizontal main channels are indicated by outer circle squares 1 ', 3', 5 ', 9' and 13 '. Vertical channels derived from the interrelationship between the five wide-range horizontal main channels or the reverberations generated or individually supplied, indicated by a dashed square 23 'in the center. The 23 wide-range output channels are indicated by circles numbered 1-23. The outer circle with 16 output channels is on the horizontal plane, and the 6 output channels on the inner circle are located at 45 degrees above the horizontal plane. Output channel 23 is placed on top of one or more listeners. The five two-input decoding modules are represented by arrows 24-28 drawn on the outer circle, and are connected to each pair on the main channel on the horizontal plane. Five additional two-input decoding modules are indicated by arrows 29-33, connecting the vertical channels to each horizontal primary channel. The output channel 21, which emphasizes the center rear channel, is derived between the output channel 21 and the main channels 9, 13, and 23 by a three-input decoding module, as depicted by arrows. In this manner, each module is associated with a spatially adjacent pair or trio of primary channels. Although the decoding module shown in FIG. 1 has three, four or five output channels, the number of decoding modules may be any number within a common sense. The output channels may be located between one or more primary channels or at the same location as the primary channels. Therefore, for example, in FIG. 1, the position of each main channel is also the position of the output channel. Two or three decoding modules share each input channel.
[0021]
As will be explained, it is an object of the present invention to provide a concept in which a reproduction processor will work for any number and arrangement of speakers. Thus, a 24 channel arrangement as shown can be used, but this is not the only number and arrangement that defines a continuously perceived convincing acoustic space in accordance with features of the present invention.
[0022]
The desire to use a large number, preferably a user-selectable number of representation channels, is communicated to the number of discrete channels and / or to the playback processor, and at least one of the 24 discrete channels described above. Questions about other information that can lead to a typical channel. Obviously, one possible means is simply to transmit 24 discrete channels. However, for content producers, having to mix different channels and having to transmit that many channels for the transmission medium not only seem annoying, but also for 24 channels. Such an arrangement is nothing more than an example, as the arrangement is merely an example. Further, it is desirable to enable transmission of 24 or more or less common expression channels from a group of common transmission signals.
[0023]
One method of reproducing output channels is to use spatial interpolation calculated using the weighted sum of each output channel, assuming that the number of channels is sufficient to use this method. However, this requires thousands to millions of transmission channels, which is similar to using a finite impulse response with hundreds of taps when performing temporal interpolation of the signal. It becomes. Reducing the number of transmission channels to a realistic number requires the use of psychoacoustic principles and effective and dynamic spatial interpolation from fewer channels, but it requires complete acoustics. It doesn't answer the question of how many channels are needed to perceive space.
[0024]
This question can be addressed by attempts made several years ago by the inventor of the present invention and recently imitated by others. Previous attempts have at least observed that conventional two-channel two-way recording is capable of reproducing a realistic right / left acoustic image spread. However, due to the incompleteness of the adopted head-related transfer function (HRTF) and the lack of consideration for the factors of head movement, the front / rear position was incorrectly specified. To circumvent such difficulties, dual-channel (4-channel) recording was employed using two sets of directional microphones, spatially balanced to the size of a human head. One set was directed towards the front and the other towards the back. The results recorded in this way were played with four speakers placed near the head to mitigate acoustic cross-coupling effects. This arrangement provided clear information in the front and rear direction from the microphones and speakers, in addition to the time and amplitude localized real sound from each set of speakers. The result is a very impressive surround sound demonstration, and the height information could not be accurately reproduced. Another recent attempt has been to add one center front channel and two height channels, thereby increasing realism by adding similar, and possibly height information. Reports have been made.
[0025]
Thus, from both psychoacoustic perspectives and experimental findings, such perceptual information is likely to be placed in the horizontal direction of four to five "binaural-like" It can be seen that this is conveyed by the channel and, in addition, possibly one or more vertically oriented channels. However, signals having binaural characteristics sent to a set of channels are not suitable for direct reproduction by a group of loudspeakers due to insufficient separation between the midrange and low range. Therefore, it is simpler to mix the output channels from nearby transmission channels, leaving the channels independent, rather than transmitting them, just for the encoder to reproduce in the decoder (as a binaural pair). Is a direct way. This not only allows for direct playback without decoders for equal numbers of speakers, or, if desired, by adding optional mixing into fewer channels with a passive matrix decoder. , At least on the horizontal plane, corresponds to the standard arrangement of the conventional 5.1 channels. This allows some time delay so that sound arriving from an intermediate direction tends to be positioned essentially at the nearest microphone (or the nearest pair of microphones when aligned horizontally). And is therefore widely compatible with natural recordings such as those taken from five directional microphones.
[0026]
In this way, from a perceived point of view, this allows the channel conversion decoder to include a standard 5.1 channel program. Then, as described above, it is possible to reproduce sound that can be understood through an arbitrary number of speakers arranged in the horizontal direction, including the case where 16 horizontal speakers are installed by 24 channels. By adding vertical channels, as sometimes proposed in digital cinema systems, the individually derived and perceptually valid signals in all of the 24 channels can be recognizable in a wide listening position, such as in a continuous acoustic space. Can be sent to Of course, if fine source channels, such as particles, are available at the encoding site, additional information from these may be used to aggressively modify the scale factor of the encoding matrix as a pre-compensation for decoder limitations. It may be possible to use it. Alternatively, it may be possible to simply add it as additional side-chain (auxiliary) information, probably similar to the coupling coordinates used for AC-3 (Dolby Digital) multi-channel coding. However, perceptually, such special information is unnecessary, and it is not preferable from a practical point of view to include such information. The channel conversion decoder used here is not limited to the operation of a 5.1-channel sound source, and may be used for a sound source of a higher sound source or a sound source of a lower sound source. It justifies the confidence that it can be obtained from the sound source.
[0027]
Here, how to develop the intermediate output channels from the few transmitted channels remains unanswered. A solution proposed as one feature of the invention is to re-use the concept of vertical acoustic image in a slightly different way. Creating a vertical acoustic image with a small group of speakers is not feasible as a performance to the audience because the listener must be approximately equidistant from each speaker. However, by introducing the concept of a virtual intermediate channel that makes it appear as a signal appearing between adjacent real output channels, a listener in an unexpected place can be made to function. Accordingly, one feature of the present invention is that it comprises a series of modularized signal interpolation processors, each of which effectively emulates a listener located at an optimal location, and which is capable of emulating an amplitude adjusted signal. We propose something that works in a manner similar to the human auditory system that extracts virtual images and outputs them to real loudspeakers. Here, it is desirable that the group of loudspeakers is arranged to have a density sufficient to fill the space between the speakers with the virtual image.
[0028]
In general, each decoding module gets its input from the closest main channel, for example, three or more main channels for a group of speakers located on top. As one method of generating an output channel based on inputs from two or more main channels, a method in which functions for each pair are collected may be employed. For example, an output from a decoding module constituting a certain pair is used as an input to another module. However, this has two disadvantages. First, decoding modules that transmit signals one after the other will incorporate the accumulated time constants, which will result in some output channels responding faster than others, adversely affecting audible position perception. . A second drawback is that simply associating each pair can only determine the output channel induced on the middle, that is, the line connecting the pairs. This disadvantage is eliminated by using more than two primary channels. Therefore, a technique has been developed in which general pairwise association is extended to associate three or more output signals. This technique is described below.
[0029]
The determination of the horizontal position (of the sound source) in the human ear is based on two localization cues: mutual amplitude and mutual time difference of arrival. The latter cues are only valid when the signal pairs are close in time, about ± 60 milliseconds. The practical effect is that, assuming a common signal content associated or nearly associated with the two real channels, an insubstantial intermediate sound image is generated at the position corresponding to the left / right amplitude difference. Just do it. (Note: Two signals can have a span of +1 to -1 in their mutual correlation coefficient. Signals with perfect correlation (correlation coefficient = 1) have the same waveform and time structure. However, the amplitude can have different values depending on the displacement of the acoustic image). As the correlation coefficient of a signal pair decreases below I, the perceived acoustic image tends to spread to two uncorrelated signals. In that case, the sound image is no longer an intermediate sound image, but merely separate left and right sound images with a distance. With a negative correlation, the normal ear feels the same as with an uncorrelated signal pair, although the distance between the two acoustic images increases. The correlation is performed on the basis of a critical band, and a band above about 1500 Hz. The signal waveform in the critical band is not used as the signal itself, but is used to secure human computing power (MIPS).
[0030]
Vertical positioning is slightly more complicated, relying on the pinnae cues of the head related transfer function (HRTF) and the cues by dynamic horizontal modulation with head movement. However, the net effect is similar to horizontal positioning, in that the amplitudes are panned, correlated, and correspond to the location and integration of the recognized acoustic image. Vertical position resolution, however, is less accurate than horizontal resolution. In addition, the density of the main channel group for securing appropriate complementary performance is not so required.
[0031]
The advantage of using a directional processor to simulate the functions of the human ear is that any imperfections or limitations in signal processing can be recognized by the system with little discrimination from the original, fully continuous speech. It is perceptually hidden, as are the imperfections and limitations of the human ear that can be tolerated.
[0032]
The present invention provides for active mixing, including reproduction with as many loudspeakers as there are input signals without decoding, no matter how many or no output channels, and with mono, stereo or Lt / Rt. It is designed to be used effectively, but the number of output channels is large, and in some sense discretionary, The number of expression channels / loudspeakers, including the existing 5.1 channel soundtrack and the next generation of 11 or 12 channel digital cinema soundtracks, equal to or less than the number of encode channels It is desirable to be able to employ those that use a number of sound sources.
[0033]
The practice of the present invention discloses four principles. Error suppression, dominant management, constant output and synchronized smoothing.
[0034]
Error suppression refers to the concept that, when there is a possibility of a decoding error, the position of each decoded sound source is a reasonable value close to the true value in the intended direction. This controls the degree to which the decoding method is changed. When more aggressive decoding is anticipated, with the potential for errors that increase the spatial variance, it is usually the case that slightly less accurate data is traded off for better spatial management. It is desirable to do coding. Even in situations where more accurate decoding can be reliably applied, if the dynamic signal situation requires switching the decoder between aggressive and normal modes, It is not advisable to do so if it can have adverse consequences.
[0035]
Dominant management is a further variant of error suppression where a sufficiently clear dominant signal should be panned by the decoder only to the nearest neighbor output channels. Requirements. This condition is necessary to preserve and maintain the integrated image of the dominant signal and contribute to the discontinuity of the matrix decoder. If a signal is dominant, another output signal, extracted from the relevant key signal or other complementary output channel matrix variables used to derive the dominant signal ("anti-dominant variables / signals") , The signal is suppressed.
[0036]
Constant output means that the sum of the decoded output powers is not only equal to the input power, but also that the input and output powers of the directional signals transmitted and encoded by each channel and the main channel group are identical. This minimizes the effects of gain pumping.
[0037]
Synchronized smoothing is applied to systems with a smoothing time constant that depends on the signal, and if any smoothing network is in the decoding module, the time constant is switched to fast mode and other modules Smoothing networks within are also switched. This prevents signals with the new dominant direction from appearing to slowly appear and disappear from the previous dominant direction.
[0038]
Best Embodiment of the Invention
Decoding module
Since encoding the direction of the sound source is predominantly based on finding the association between adjacent major channels, transforming and decoding the channels is particularly useful for transmitting each of them in a bend that approximates the human ear. It is based on the use of a series of semi-autonomous decoding modules that reproduce the intermediate output channel, which forms part of the channel, in a general sense the output channel.
[0039]
The operation of the decoding module is as functionally similar as possible to the human ear, correlates with the nominal current main direction and determines the relative width of the image by using the amplification factor Is performed based on the combination of
[0040]
The arithmetic processing device extracts the audio signal of the output channel using the control information derived from the amplification coefficient and the mutual correlation. Since this is a linear basis limitation, to avoid distortion components, the decoder forms a set of weighted sums of the primary channels containing the signals of interest. (As described below, it is preferable to include the information of the non-neighboring primary channels in the calculation of the weighted sum.) The formation of this limited but dynamic interpolation is more generally performed in a matrix. Entrusted. If, at the source, the signal sought is associated (selectively amplified) with the nearby M primary channels, then the problem is the M: N matrix. In other words, the output channel is relatively harmonious with the input channel.
[0041]
In particular, in the case of a two-input decoding module, an active 2: N matrix decoder, such as the now classic Dolby Pro Logic matrix decoder, having a pair of decoding modules corresponding to the Lt / Rt encoded signals. Very similar to the issues considered.
[0042]
Note: The output from the 2: N matrix decoder is sometimes referred to as the primary channel. However, in this specification, the term "primary" is used when referring to the input channel of the decoder transmitting the channel.
[0043]
However, there is at least one distinct difference between the prior art active 2: N decoder and the processing of the decoding module according to the invention. The former uses left / right amplification on the assumption of a channel transmission decoder to indicate the left / right position, and also indicates the sum of the Lt / Rt encoded channels versus the deviation to indicate the front / back position. The phase between the channels is used which is clearly dependent on the ratio.
[0044]
There are two problems with processing by such an active 2: N decoder. One is, for example, a signal that is perfectly correlated with (front) but off-center, resulting in a sum-to-deviation ratio that is less than infinity, and accurately describes the position of the front. Gone. (It looks like a signal on the back surface with an inverse correlation off center). The result is a somewhat distorted decoding space. The second drawback is that the positional relationship is many-to-one, which inherently introduces errors during decoding. For example, in a 4: 2: 4 matrix system, a pair of Left-In Right-In signals that do not include Front-In and Rear-In signals and have no correlation with each other are left-in and right-in signals. Similar to a pair of Front-In and Rear-In signals that do not include a signal and have no correlation with each other, the function is a function for an uncorrelated Lt / Rt pair, that is, a four-input function that has no correlation. When faced with an uncorrelated Lt / Rt pair, the decoder has no choice but to relax the matrix, use a passive matrix and distribute the speech to all output channels. This makes it impossible to simultaneously decode a signal constellation of only Left-Out / Right-Out or a signal constellation of only Front-Out / Rear-Out.
[0045]
The potential problem is that using phase between channels for coding front and back positions in an N: 2: N matrix system behaves in a manner that is contrary to the behavior of the human ear, which does not use phase to determine front and rear positions. That is to put it. The present invention provides for at least three non-collinear principals to represent front and rear positions by assuming principal channel directions without assigning direction differences depending on relative phase or polarity. Works well with channels. Thus, the uncorrelated or inversely correlated dominant transmitted signal pairs are unambiguous, have no intermediate signals, show no "reverse" direction, and have independent main output Decoded into a channel signal. (By the way, this avoids an inappropriate "center pile-up" phenomenon in an active 2: N decoder. In this phenomenon, the decoder uses the uncorrelated Left-In and Right-In signals. Since the sum and deviation are sent to the center and surrounding channels, the uncorrelated Left-In and Right-In signals are expressed with weaker separability.) Of course, 2: N decoder, N = 4 or 5, it is possible in principle to spatially extend the Lt / Rt signal pair by connecting stepwise to an N: M channel conversion system. However, in this case, all restrictions such as center pile-up in the 2: N decoder are carried over to the increased output channels. Also, these functions can be incorporated in a channel conversion decoder configured to accept two-channel Lt / Rt signals. In this case, it is possible to modify the operation so that the signal having the inverse correlation is interpreted as having the backward direction, without significantly impairing other processing procedures. However, even in this case, since there are only two transmitted signals, ambiguity remains in the decoding.
[0046]
Thus, each decoding module, especially one with two input channels, has an arbitrary output channel and is similar to prior art active 2: N decoders, in which front-to-back detection is impossible or modified. . Of course, uniquely deriving a larger number of channels from a small number of channels using a matrix is like solving M unknowns with N linear equations when M is larger than N, and mathematically Impossible. Thus, it can be expected that the decoding module will sometimes not be able to make less than a perfect reconstructed channel in the presence of multiple source direction signals. However, the human hearing system is limited to the use of two ears and tends to have to follow the same restrictions, but it is a system that operates all channels but recognizes them as separate. . Independent channel quality is something that must be considered in order to be able to mute other channels and still accommodate listeners who may be near one speaker.
[0047]
Indeed, the ear works on a frequency-dependent basis. However, as many audio images have similar correlations at all frequencies, as a wideband system, as demonstrated empirically by the Pro Logic decoder, a wideband channel conversion system is a satisfactory result in some applications. Is expected to be obtained. It is also possible to decode multiple channels by using a similar process on a band-by-band basis, and in each case using the same encoding signal, where the number and bandwidth of each band is left to the developer. It is left as a free parameter. Although higher MIPS seems to be required than for wideband processing, the computational burden is not very large if the input signal is divided into data blocks and the processing is performed for each block.
[0048]
Before describing useful algorithms by the decoding module in the present invention, shared nodes will be described.
[0049]
Shared Nodes
If all the main channels used for the decoding module are independent, the decoding module itself is also autonomous and independent. However, this usually does not occur. Any transmitted channel generally shares a separate output signal with two or more adjacent primary channels. If independent decoding modules were used to decode a group of signals, each of which would be affected by signals from adjacent channels and could result in significant decoding errors. In fact, the two output signals of the adjacent decoding modules will be pulling on each other, since the level of the common primary junction with both signals will increase. If, in the usual case, the signal is dynamic, a number of interactions can result in a dynamic error depending on the signal having undesirable properties. This problem does not occur with Pro Logic or other active 2: N decoding, since a single independent pair of channels is used as the input to the decoder.
[0050]
Thus, compensation for the "Shared Node" effect is needed. One possible way to do this is to share the common node and subtract the recovered signal from the common node before trying to recover the output signal of the neighboring decoding module. This often fails with fallback. That is, each decoding module estimates the common output signal energy present in the input channel, and the supervisor routine communicates the output estimate of the module adjacent to each module to each module.
[0051]
Calculation of shared energy in pairs
For example, assume a primary channel pair A / B that includes a common signal X in addition to independent signals Y and Z.
[0052]
A = 0.707X + Y
B = 0.707Y + Z
Here, the scale factor 0.707 = √0.5 gives a multiplier for holding a function for an adjacent main channel.
[0053]
(Equation 1)

Because there is no correlation between X and Y
XY = 0
Therefore,
(Equation 2)

This is because there is no correlation between X and Y, and the total energy of the main channel A is the sum of the energies of the signals X and Y.
[0054]
Similarly,
[Equation 3]

Since X, Y and Z are uncorrelated, the cross product of A and B is
(Equation 4)

[0055]
Thus, for the case of an output signal equally shared by two adjacent primary channels that may contain independent and uncorrelated signals, the average of the cross product of the signals is the common signal component in each channel Equal to the energy of If the common signals are not equally shared, that is, if they are biased toward one of the primary channels, the average of the cross product values will be the geometric mean of the energies of the common components in A and B. From A and B, an estimated value of a common component in each channel is obtained by normalizing the amplification coefficient of the channel by the mean square. The actual time average is calculated by a leaky integrator with an optimal decay time constant to reflect the state continuously. The smoothing by the time constant can be performed more precisely by giving options to the nonlinearity and the decay time. In a multi-band system, the measurement may be performed based on the frequency.
[0056]
Calculation of higher order common energy
Deriving the common energy of a decoding module with more than two inputs requires calculating the average of the cross-product values of all the input signals. If the calculation is simply performed for each pair of inputs, it becomes impossible to distinguish separate output signals between each pair of inputs and the entire common signal.
[0057]
For example, consider three main channels, A, B and C, consisting of uncorrelated signals W, Y, Z and a common signal X.
[0058]
A = X + W
B = X + Y
C = X + Z
When the average cross product value is calculated, the term that is a combination of W, Y, and Z is canceled, and X ³ The average value of remains.
[0059]
(Equation 5)

[0060]
However, as expected, if the time average of X is zero, its cubic average is also zero. X that is positive for all non-zero values of X ² Is different from averaging ³ Has the same sign as X, so the positive and negative contributions are cancelled. Obviously, this is true where X is an odd power, corresponding to an odd number of inputs to the module. However, even if the index is 2 or more, the result will be wrong. For example, in the case of four inputs in which the components are (X, X, -X, -X), the average cross product value is the same as in the case of (X, X, X, X).
[0061]
This problem has been solved by changing the method for calculating the average value of the products. Before averaging, the sign of each product is discarded by taking the absolute value of the product. The sign of each term in the product is examined. If all the signs are the same, the absolute value of the product is used for averaging. If either sign is different from the others, the negative value of the absolute value of the product is averaged. Since the number in which the same code combination can occur and the number in which a different code can occur are different, to compensate for this, a weight dispute consisting of the ratio of the number of different code combinations to the number of the same code combinations, Applies to negate the absolute value of the product. For example, in a three-input module, when the combination having the same sign is subtracted from the combination of six different signs from the combination of eight, there are two combinations, and as a result, the scale factor becomes 2/6 = 1/3. . This correction results in a positively increasing integrated or summed product only if all inputs of the decoding module have a common input signal component.
[0062]
However, they must all have the same order in order to make the mean values of the modules of different series comparable. Conventional second-order correlation results in a quantity having the product of two inputs or energy or power dimensions. Therefore, the items that average in higher order correlations must be modified to have the power dimension. Therefore, for correlations of order kth, the absolute value of each product must be increased to a power of 2 / k before averaging.
[0063]
Of course, without considering the order, the energy of each input node of the module can be calculated as the root mean square of the signal of the corresponding node if necessary. Then, it is not necessary to first increase the order to kth power and then decrease to the second order value.
[0064]
Shared Nodes: Neighbor levels
By using the mean square value and the modified cross product value of the primary channel signal, the amount of signal energy in the common output channel can be calculated. Although the above example requires a primary interpolation process, if one or more nodes in A / B (/ C) have a common signal component, which has no correlation with any other signal If they were in common with other modules like this, the mean of the above cross product would not be affected, since the calculation is essentially not affected by any image stretching effects (note: the two output signals are uncorrelated). If not, they have a tendency to stretch some decoders, but the effect on the human ear is similar, so the operation of the system remains true to human hearing.)
[0065]
Once each decoding module has calculated an energy estimate for the common output channel signal at its primary node, the monitoring function transfers the common energy to adjacent modules. Then, at the point of the common energy, the signal of the output channel can be extracted in the following process. The calculation of the common energy used by a module at a node takes into account the hierarchical structure that may overlap with modules of different orders and calculates the common energy of any lower-order module sharing the same node. , It is necessary to subtract the common energy of higher order modules.
[0066]
For example, as signal energy X ² In addition to A and B, which represent two horizontal directions in two adjacent main channels, a main channel C which represents a vertical direction, and further toward the inside (that is, the restrictions of A, B and C). ), And consider the presence of intermediate or derivative output channels. The common energy of the three input modules, along with the inputs (A, B, C), is X ² It becomes. However, the two input modules (A, B), (B, C) and (A, C) are also the same. If the modules (A, B, C), (A, B) and (A, C) related to A are simply added, X ² 3X without becoming ² Is the result. Higher order modules are first subtracted from the common energy estimates of the overlapping lower order modules so that the common node energy calculation is correct. Therefore, the common energy X of the higher order modules (A, B, C) ² Is subtracted from the calculated value of the common energy at the two input modules, each resulting in 0, and the common energy at node A is X ² + 0 + 0 = X ² Is calculated.
[0067]
Extract output channel signal
As noted above, the process of recovering the aggregate of the output channels from the transmitted channels in a linear manner basically forms a matrix that is the sum of the weighted primary channels to derive the output channels. That is. The optimal choice of the scale factor in the matrix generally depends on the signal. In fact, if the number of currently active output channels is the same as the number of channels transmitted (but representing different directions), the system will be precisely determined and mathematically effective The matrix can be decoded, and a signal independent of the signal source is reproduced. Even when the number of active output channels is greater than the number of primary channels, it is possible to calculate a pseudo inverse transform matrix.
[0068]
Unfortunately, there is a problem with this approach. In this approach, the computational burden of achieving high-precision floating-point, especially on multi-band bases, is not a trivial matter. Although the intermediate signal is assumed to be directed to the nearest dominant channel, the mathematical inversion, or simulated reading of the effective matrix, is generally due to node sharing effects, With a contribution from every primary channel to each output channel. If there are imperfections in decoding, and that is, in fact, unavoidable, the primary channel can be reproduced from a spatially separated output channel, which is never Not desirable. In addition, the calculation of the quasi-inverse transform tends to lead to a minimum-RMS-energy solution, which minimizes the separation and spreads the sound as much as possible. It is contrary to the original intention.
[0069]
Therefore, in order to realize a decoder that is practical and accepts that a spatial decoding error is intrinsic, the same module configuration used for detecting a signal is adopted for signal reproduction.
[0070]
The following are details of the reproduction procedure in which the output signal is reproduced by the decoding module. Here, the effective position of each output channel leading to the module is needed to direct the signal to a physical position, ie to calculate the ratio of the effective matrix encoding coefficients associated with that direction. Note that the amplification factor is assumed to be indicated by: To avoid the problem of division by zero, the coefficients are calculated as the ratio of the matrix coefficients of one channel to the root mean square (RMS) of the matrix coefficients of the input channel (usually one). For example, in a two-input module with L and R inputs, the energy ratio used is the energy of L to the energy of L and R ("L-ratio"), which behaves between 0 and 1. Good thing. If the two-input decoding module has (1.0, 0), (0.89, 0.45), (0.71, 0.71), (0.45, 0.89) and (0, 1) .0) has five output channels consisting of pairs of encoding matrices, since each scale factor pair has a root mean square value of 1.0, the corresponding L-ratios is 1.0, 0 .89, 0.71, 0.45 and 0.
[0071]
The signal energy of each input node (primary channel) of the decoding module is subtracted by the request from the neighboring decoding module of all signal energies that share the node, and the result is normalized to the remaining calculations. Input signal level.
[0072]
The dominant direction index is calculated as the vector sum of the dominant direction and weighted by relative energy. For a two-input module, this simplifies to the normalized input signal level, L-ratio.
[0073]
The output channel that separates the dominant direction is determined by comparing the L-ratio of the dominant direction in step 2. For example, if the input L-ratio of the 5-output decoding module is 0.75, then 0.89>0.75> 0.71, and the second and third output channels are the main signal directions. .
[0074]
A panning scale factor that associates the dominant signal with the most recent partitioned channel is calculated from the ratio of the channel to the anti-dominant signal level. Combined with a special output channel, this anti-dominant signal is a matrix in which the input signal of the corresponding decoding module is combined with the scale factor of the output channel anti-dominant matrix. It is a signal that sometimes comes up as an answer. The scale factor of the anti-dominant matrix from the output channel is the root mean square sum (RMS sum) such that one dominant signal has zero output when sent to the corresponding output channel. It is a scale factor including 1.0. Here, if the scale factor of the encoding matrix of the output channel is (A, B), the anti-dominant scale factor of this channel is exactly (B, -A).
[0075]
Proof
If one dominant signal is sent to an output channel with an encode scale factor (A, B), the signal is amplified to (kA, kB). Here, k is a signal amplification coefficient as a whole. Therefore, the anti-dominant signal of this channel is (kA * B-kB * A) = 0.
[0076]
Thus, if the dominant signal is the input signal (x (t), y (t)) of a two-input module with an input amplification factor normalized so that root mean square (RMS) = 1. , The expanded main signal (dominant signal) is dom (t) = Xx (t) + Yy (t). If the position of the signal is partitioned as output channels with matrix scale factors of (A, B) and (C, D), respectively, the scaling dom of the dominant signal with respect to the former channel is scaled with respect to the former channel. (t) is:
SF (A, B) = sqr ((DX-CY) / ((DX-CY) + (BX-AY)))
On the other hand, the scale factor of the dominant signal corresponding to the latter channel is:
SF (C, D) = sqr ((BX-AY) / ((DX-CY) + (BX-AY)))
Since the dominant direction is sent from one output channel to another, these two scale factors move in opposite directions from zero to one while keeping the total power constant.
[0077]
An anti-dominant signal is calculated and sent to non-dominant channels, with appropriate gain scaling. An anti-dominant signal is a matrix signal from which any dominant signal has been removed. If the input to the decoding module is (x (t), y (t)) with normalized amplitude (X, Y), the non-dominant output channels Regardless of position, the dominant signal is dom (t) = Xx (t) + Yy (t), and the anti-dominant signal is dom (t) = Yx (t) -Xy (t).
[0078]
In addition to the dominant / anti-dominant signal distribution, it is basically the scale factor of the output matrix already discussed, and is a passive factor with a scaling factor to preserve power. Using the matrix, a second signal distribution is calculated.
[0079]
The correlation between the input signals of the decoding module is calculated by dividing the average of the cross product values of the input signals by the square root of the normalized input level.
[0080]
Returning to the expansion process, the final output signal is weighted in the distribution of the dominant passive signal using the correlation coefficient of the input signal of the decoding module to derive the crossfade factor. It is calculated as the sum of the crossfade. When the correlation coefficient = 1, the distribution of the main signal / anti-dominant signal is exclusively used. As the correlation coefficient decreases, the group of output signals is spread to a passive configuration by crossfading, leading to a small positive correlation. This value depends on the number of output channels connected to the decoding module, but is typically between 0.2 and 0.4. As the correlation coefficient further decreases toward zero, the passive amplitude power distribution gradually decreases, reducing the level of the output channel while simulating the human ear for such signals.
[0081]
Vertical processing
Most of the processing described so far has been applied to developing an output channel from an adjacent main channel without considering the output and the direction of the main channel. However, because the ears are placed horizontally, the correlation between channels is less sensitive to vertical than horizontal. To maintain the fidelity of the function at the ear, relax the correlation of the constraints on the interpolation of the processing equipment used for vertical inputs, such as processing the correlation signal first applying a warp function It would be desirable. However, this does not mean that using the same processing as for the horizontal channel does not adversely affect hearing. Using the same process would simplify the overall decoder configuration.
[0082]
Strictly speaking, the vertical information includes audio from both the top and bottom. Then, the configuration of the decoder operates in the same manner in both cases. However, in practice, usually only a little natural sound comes from the bottom. Thus, processing and channels for such signals from below may be omitted without compromising the spatial recognition fidelity of the system.
[0083]
This concept has practical significance for the conventional application of channel conversion to 5.1 channel surround, although the vertical channel is missing, of course. However, it can also include vertical information, such as a flyover effect that outputs across many or all horizontal channels. Therefore, it is possible to develop a virtual vertical channel from such a signal source by searching for a correlation between non-adjacent channels or groups of channels. In the presence of such correlations, they represent the presence of vertical information from above, rather than from below the listener. In some cases, it is possible to derive virtual vertical information from the reverberation generator, perhaps by harmonizing it with the environment of the target listener. Once a virtual vertical channel is developed or derived from a 5.1 channel source, a large number of channels, such as the 24-channel arrangement described above, can be used as if the vertical channels were provided. Deployment becomes possible.
[0084]
Directional memory
One aspect is that the control of the decoding module described above is similar to an active 2: N decoder such as a Pro Logic decoder. That is the point that in its processing only the memory is in the smoothed network which derives the basic control signal. At any one time, there is only one dominant direction and the value of the input correlation, and these values are used to process the signal expansion.
[0085]
However, in a particularly complex acoustic environment (such as a typical cocktail party), the human ear has been shown to be accustomed to a certain location memory or location, so that a clearly identified location Is recognized as a different sound, that is, a position that cannot be clearly identified compared to the general direction recognized when coming from the same sound source.
[0086]
Simulating this effect in the decoding module (even in Pro Logic decoding) adds an obvious mechanism to keep track of the latest main direction, and in the middle of directionally confusing signal situations, This is possible by weighting the distribution of the output signal toward the direction. This makes it possible to enhance the separability and stability of a complex signal group in recognition of sound reproduction.
[0087]
Correction of signal correlation and selection of channel mixing
As described, the spreading decision for each decoding module is based on the cross-correlation of the input signal. This may underestimate the amount of output signal content under certain conditions. This can occur, for example, in a way that the arrival times are slightly different, do not go to the center and are not of the same amplitude, and result in lower correlation values for natural signals. This effect is accentuated when the delay between the channels is correspondingly extended and the microphones are used at widely spaced intervals. To compensate for this, the burden of slightly increasing the required MIPS for processing is incurred, but the correlation calculation may be extended to cover the range of time delay between channels. Furthermore, since the nerve cells in the auditory nerve have an effective time constant of about 1 msec, a more realistic correlation value can be obtained by the primary smoothing action of the voice shaped by a smoothing circuit having a time constant of 1 msec. Is also good.
[0088]
Furthermore, if the content producer has an existing 5.1 channel program with almost uncorrelated channels, the spread equivalence when processed by the channel conversion decoder is improved by slightly mixing adjacent channels. However, this increases the correlation and allows the channel conversion decoding module to spread more evenly between the intermediate output channels. Such mixing can be done selectively, for example, to keep the signals of the channels in front of the center unmixed in order to keep the dialog track compact.
[0089]
Loudness compression and expansion
The encoding process involves mixing a large number of channels into a small number of channels, which indicates that without some form of gain compensation, the encoded signal may be omitted. This problem also exists in the conventional matrix encoding. However, channel conversion is potentially a bigger problem due to the larger number that is mixed and given to the output channels. In such a case, the overall gain scale factor is derived from the encoder and transferred to the coded bitstream of the decoder to avoid signal omission. Normally, this value is 0 dB, but to avoid signal omission, the decoder may provide an equivalent amount of gain correction and the encoder may provide a non-zero attenuation value.
[0090]
If a decoder were to be used to perform this process on an existing multi-channel (eg, an existing 5.1-channel soundtrack) without such a scale factor program, the expected values (0 dB and A fixed scale factor with estimation) may optionally be used, or an extension based on signal level and / or dynamics may be applied. Alternatively, valid metadata, such as values that normalize the dialogue, could be used to adjust the decoder gain.
[0091]
The invention and its various aspects may be embodied as analog circuits or, more likely, as software functions formed by digital signal processors, general purpose digital computers incorporating programs, and / or special purpose digital computers. You. The signal flow interface between analog and digital signals may be implemented with functions and / or firmware in suitable hardware and / or software.
[Brief description of the drawings]
[0092]
FIG. 1 is a schematic plan view showing an idealized arrangement of decoders.

Claims (8)

An audio channel conversion method for converting M audio input channels representing an acoustic space into N audio output channels representing the same acoustic space, wherein each channel is a sound of one sound representing a sound from a certain direction. Flow, M and N are positive integers, and M is at least 2 and
Generating one or more sets of output channels, each set having one or more output channels, each set being associated with two or more spatially adjacent input channels, and each output channel in the set being A method for determining the correlation of the two or more input channels and a level of interaction of the two or more input channels. The audio channel conversion method according to claim 1, wherein
An audio channel conversion method comprising having a set of output channels associated with two input channels. The audio channel conversion method according to claim 1, wherein
An audio channel conversion method, wherein one or more sets of said output channels are associated with three or more input channels. The audio channel conversion method according to claim 1, wherein
The set of one or more output channels is associated with more input channels than the number of input channels with which the other set of one or more output channels is associated;
The audio channel conversion method, according to a hierarchical order in which each set is ranked according to the number of input channels associated with the output channels with a higher ranking as the number of input channels increases. Determining the correlation of the input channel associated with the set of output channels,
An audio channel conversion method, wherein the progress of the processing procedure is determined according to a hierarchical order. The audio channel conversion method according to claim 4, wherein
The method is characterized in that a higher order set of processing results is considered. The audio channel conversion method according to claim 1, wherein
An audio channel conversion method, characterized in that the determination of the means for determining the correlation between two or more input channels and the determination of the level of interaction between the two or more input channels are performed in the frequency domain. The audio channel conversion method according to claim 1, wherein
The method according to claim 1, wherein the method employs a non-linear time constant. 9. The audio channel conversion method according to claim 1, wherein the input channel has input channels indicating three or more directions that are not on the same straight line.