JP3715332B2 - Communication system, receiving apparatus and method - Google Patents

Description

【００７０】
【表２】

【００７１】
次に、ネットワークとしてサービス総合ディジタル網（ＩＳＤＮ:integrated service digitial network) の１Ｂ（６４Ｋ）を用いた場合の性能は表３のようになる。なお、ＩＳＤＭを用いた場合には、画像データを圧縮しないと最小の画面サイズ（８０×６０）においてもフレームレートが０．２（fps)となり、実用的でないため、深さ１６ビットの無圧縮画像の転送性能の評価は省いた。
【００７２】
【表３】

【００７３】
このような情報処理装置を用いることにより、画像と音声の取り込みや圧縮伸張、またネットワークプログラミングを意識することなく容易にマルチメディアアプリケーションが作成可能となった。
【００７４】
４．４ メディア間同期の実現
この情報処理装置では同期をとりメディアデータは、単一のストリームにインターリーブして転送している。また、ネットワークプロトコルとしてＴＣＰ／ＩＰを用いているため、パケットの順序は保証される。さらに、複数のメディアデータの取り込みを同時に行い、インターリーブしてデータ送信時に同期を保証できれば、データ受取側でも複数のメディアデータの同期はとれているものと仮定できる。実際この方法で単一ネットワーク上にて転送した音声と動画のメディアデータの同期は満足のいく結果となった。
【００７５】
４．５ 遅延との関係
上述の実装では、ネットワークプロトコルとしてオペレーティングシステムのＵＮＩＸで標準的なＴＣＰ／ＩＰを用いている。ここで、図８に示すように、メディアデータを直接送ることの出来る環境では、送信側と受信側のバッファの制御を情報処理装置が行うことができ、細かな流量制御が行える。
【００７６】
以下、図１に戻って、上述したようなことを具体的に実現する本発明実施例の通信システムについて説明する。
【００７７】
図１において、マイクロホン１やビデオカメラ２からの音声信号や映像信号（動画信号）は、それぞれ対応するＡ／Ｄ変換回路６５，６６に送られて、Ａ／Ｄ変換される。Ａ／Ｄ変換回路６５からの音声データは、音声データ圧縮回路６７に送られ、ここで前述したように音声の圧縮方式として例えばＧ．７１１，Ｇ．７２１，Ｇ．７２２，Ｇ．７２８などのうちのいずれかを用いた圧縮が施される。また、上記Ａ／Ｄ変換回路６６からの映像データは、後述のフレームバッファ６８を介して映像データ圧縮回路６９に送られ、ここで前述したように動画の圧縮方式として例えばＪＰＥＧ，Ｈ．２６１，ＭＰＥＧなどのうちのいずれかを用いた圧縮が施される。
【００７８】
これら圧縮回路６７，６９からの圧縮された音声データと映像データは、マルチプレクサ５によってマルチプレクスされ、その後データバッファメモリ（例えばＦＩＦＯメモリ）６を介してネットワーク送信部７に送られ、当該ネットワーク送信部７から例えばイーサネットなどのネットワーク８を経て、受信側９１のネットワーク受信部１１に送られる。
【００７９】
また、この図１の通信システムにおいては、入力端子８９には上記音声や動画を除く信号として例えばテキストデータやプログラムデータ，数値データ，他のバイナリデータ等が供給される。これらデータは、データ処理回路９４において例えば誤り訂正符号が付加される等の処理が施された後、マルチプレクサ５を介し、さらにネットワーク送信部７から出力されるようにもなっている。
【００８０】
当該受信側９１のネットワーク受信部１１で受信したデータは、分離回路１２によって上記圧縮された音声データと映像データとに分離され、それぞれ対応する音声データバッファメモリ（ＦＩＦＯメモリ）１３と映像データバッファメモリ（ＦＩＦＯメモリ）１４に送られる。
【００８１】
また、バッファメモリ内容量測定回路２０は、映像データバッファメモリ１４の書き込みアドレス及び読み出しアドレスから当該メモリ１４の内容量の測定を行い、その測定結果（後述する測定値Ｖ）を演算判断回路８１に送る。当該演算判断回路８１は、上記測定回路２０からのメモリ１４の内容量測定結果から上記映像データバッファメモリ１４の内容量が上記上限（後述するＶｍａｘ）を越えたことが検出されると、その旨の信号を制御信号としてネットワーク送信部７３を介して送信側９１に送る。
【００８２】
さらに、上記演算判断回路８１では、上記測定回路２０からのメモリ１４の内容量測定結果（後述する測定値Ｖ）から映像データバッファメモリ１４の内容量が上記下限（後述するＶｍｉｎ）を下回ることが検出されると、映像が途切れるおそれがあるので、その旨（下限を下回った旨）の信号を制御信号としてネットワーク送信部７３を介して送信側９０に送る。
【００８３】
ここで、上記バッファメモリ内容量測定回路２０及び演算判断回路８１においては、図９に示すようなフローチャートによって、映像データバッファメモリ１４の内容量を測定し、上記制御信号を生成するようにしている。
【００８４】
この図９において、ステップＳ１１では、映像データのバッファメモリ１４の内容量の測定値Ｖ（単位は再生時間）を求める。なお、上記測定値Ｖは以下の式で求める。
Ｖ＝（バッファ読み出しアドレス）−（バッファ書き込みアドレス）
【００８５】
次のステップＳ２では、当該測定値Ｖと映像データのバッファメモリ１４の内容量の適正範囲の上限値ＶｍａｘとからＶ＞Ｖｍａｘを判定し、ノーと判定した場合にはステップＳ３２に進み、逆にイエスと判定した場合にはステップＳ３５に進む。
【００８６】
上記ステップＳ３５では、送信側９０に対して、Ｖ＞Ｖｍａｘである旨を表す信号を制御信号として送り、その後ステップＳ１１に戻る。
【００８７】
また、上記ステップＳ１２でノーと判断されてステップＳ３２に進むと、このステップＳ３２では、上記測定値Ｖと映像データのバッファメモリ１４の内容量の適正範囲の下限値ＶｍｉｎとからＶ＜Ｖｍｉｎを判定する。当該ステップＳ３２で、ノーと判定した場合にはステップＳ１１に戻り、イエスと判断した場合にはステップＳ３６に進む。
【００８８】
ステップＳ３６では、送信側９０に対してＶ＜Ｖｍｉｎである旨を表す信号を制御信号として送り、その後ステップＳ１１に戻る。
【００８９】
なお、映像データのバッファメモリ１４の内容量の適正範囲の適正値をＶｓｔａとすると、上記上限値Ｖｍａｘと上記適正値Ｖｓｔａと下限値Ｖｍｉｎとの関係はＶｍｉｎ＜Ｖｓｔａ＜Ｖｍａｘとなる。
【００９０】
上記受信側９１からの制御信号は、送信側９０のネットワーク受信部７２にて受信され、取り込み周期指定回路７１に送られる。当該取り込み周期指定回路７１は、上記制御信号に基づいて、フレーム取り込みクロック発生回路７０を制御する。当該フレーム取り込みクロック発生回路７０は、上記取り込み周期指定回路７１からの信号を受けると、フレームバッファ６８においてコマ落とし数を可変させるようなフレーム取り込みクロックを発生する。ここで例えば、コマ落としを行う（若しくはコマ落とし数を増やす）ことで、送信する映像データの量を減らすことができき、コマ落としを行わないか若しくはコマ落とし数を減らすことで送信する映像データの量を増やすことができる。
【００９１】
すなわち、上記送信側９０では、上記受信側９１からの制御信号によって図１０に示すようなフローチャートの処理を行う。
【００９２】
この図１０において、ステップＳ４１では受信側９１からの制御信号を受ける。
【００９３】
次のステップＳ４２では、受信した制御信号がＶ＞Ｖｍａｘを表す信号か否かを判断し、当該Ｖ＞Ｖｍａｘを表す信号であると判断した場合には、ステップＳ４３に進み、否と判断した場合にはステップＳ４４に進む。
【００９４】
ステップＳ４３では、上記取り込み周期指定回路７１が、フレーム取り込みクロック発生回路７０に対して、フレームバッファ６８におけるフレームの取り込みクロックの周期をＰだけ長くさせるように制御する。このステップＳ４３の後はステップＳ４１に戻る。なお、このＰの値は、例えば１％とする。
【００９５】
一方、上記ステップＳ４２においてノーと判断された場合のステップＳ４４では、上記受信した制御信号がＶ＜Ｖｍｉｎを表す信号か否かを判断し、当該Ｖ＜Ｖｍｉｎを表す信号であると判断した場合には、ステップＳ４５に進み、否と判断した場合にはステップＳ４１に戻る。
【００９６】
ステップＳ４４では、上記取り込み周期指定回路７１が、フレーム取り込みクロック発生回路７０に対して、フレームバッファ６８におけるフレームの取り込みクロックの周期をＰだけ短くさせるように制御する。このステップＳ４４の後はステップＳ４１に戻る。なお、このＰの値は、例えば１％とする。
【００９７】
これにより、本実施例の通信システムにおいては、受信側９１の映像データバッファメモリ１４の内容量が多くなって当該映像データバッファメモリ１４における入力から出力までの時間遅れ及び時間のずれが適正な値を越えるようになっても、当該時間送り及び時間のずれを適正な値に回復することができ、また、映像データバッファメモリ１４の内容量が少なくなって映像が途切れるおそれがでてきても、当該映像の途切れを防ぐことができるようになる。
【００９８】
また、受信側９１において、音声データバッファメモリ１３や映像データバッファメモリ１４から読み出された音声データと映像データは、それぞれ対応する伸張処理及びＤ／Ａ変換を施す伸張Ｄ／Ａ変換回路１８，１９に送られる。これら伸張Ｄ／Ａ変換回路１８，１９では、前記Ａ／Ｄ変換圧縮回路３，４での各圧縮処理に対応する伸張処理がそれぞれ施され、その後Ｄ／Ａ変換して出力する。
【００９９】
上記伸張Ｄ／Ａ変換回路１８からの音声信号はスピーカ２７に送られ、上記伸張Ｄ／Ａ変換回路１９からの映像信号はモニタディスプレイ２４に送られる。
【０１００】
また、上記図１の構成では、送信側９０において映像のコマ落とし数を可変することで、上記映像データバッファメモリ１４における入力から出力までの時間遅れ及び時間のずれを適正な値に回復したり、映像の途切れを防ぐようにしているが、例えば、送信側９０から受信側９１に送る画像の画質を選択することでも同様のことを行うことができる。
【０１０１】
すなわち、この画質を選択する場合には、例えば、フレームバッファ６８において取り込む各画素データを間引くことで送信するデータ量を減らしたり、逆に間引きを行わないようにすることで画質を確保する。この場合、受信側９１では、画素データを間引いたときには、例えば補間等の処理を行って画像を復元する。
【０１０２】
また、この図１の通信システムにおいて、受信側９１の上記分離回路１２に供給された受信信号は、当該分離回路１２を介して前記検出回路２５にも送られる。当該検出回路２５では、上記受信信号から音声及び動画の信号を除く信号として例えばテキストデータやプログラムデータ，数値データ，他のバイナリデータのような信号を検出する。
【０１０３】
この検出回路２５において上記音声や動画の信号を除くデータを検出すると、当該検出回路２５からは、上記分離回路１２に対して検出信号が出力される。上記分離回路１２は、上記検出回路２５からの検出信号が供給されると、上記受信信号から当該検出信号に応じたデータのみを分離して、データ処理回路２６に送る。
【０１０４】
当該データ処理回路２６では、供給されたデータに対して所定の処理を施した後、出力端子から出力する。なお、上記所定の処理としては、例えば誤り訂正処理等を挙げることができる。
【０１０５】
このように、本実施例の受信側９１においては、上記検出回路２５を有し、上記分離回路１２が当該検出回路２５からの検出信号に応じて、受信信号から音声や動画を除くデータを分離することによって、これら音声や動画を除くデータに応じた処理を行うことが可能となっている。
【０１０６】
また、本実施例の受信側９１の分離回路１２において分離された音声と映像のデータも、上記検出回路２５に送られる。
【０１０７】
このときの上記検出回路２５では、音声データに対しては例えば人間の聴覚特性を考慮して当該音声データから低域の部分を検出したり、人間の声の部分のように特に重要な音声部分を検出したりする。また、映像データに対しては例えば人間の視覚特性を考慮して当該映像データから特に視覚的に重要な映像部分を検出したり、映像内容で特に重要な部分を検出したりする。この検出回路２５からは、上記演算判断回路８１に対して上記映像データと音声データの各々に対応する検出信号が送られる。
【０１０８】
上記演算判断回路８１では、上記検出回路２５からの音声データに対応する検出信号を受けると、音声データバッファメモリ１３に対して上記検出信号に対応した音声部分が破棄されないような制御を行う。すなわち、上記検出信号に対応する音声の重要な部分については、上記音声データバッファメモリ１３からそのまま後段の音声データの伸張Ｄ／Ａ変換回路１８に送られるような制御を行う。一方、これら重要な部分以外の音声のデータに対しては、バッファメモリ１３の内容量が例えば適正値を越えた場合に破棄することで当該バッファメモリ１３の内容量の適正化を図っている。
【０１０９】
これによっても、本実施例の通信システムの受信側９１では、上記バッファメモリ１３の入力から出力までの時間遅れを適正値に回復させることができ、また、上記音声の重要部分については破棄されないために、後のスピーカ２７から出力される音声は、高品質を保つことができるようになる。
【０１１０】
一方、演算判断回路８１では、上記検出回路２５からの映像データに対応する検出信号を受けると、映像データバッファメモリ１４に対して上記検出信号に対応した映像部分が破棄されないような制御を行う。すなわち、上記検出信号に対応する映像の重要な部分については、上記映像データバッファメモリ１４からそのまま後段の映像データの伸張Ｄ／Ａ変換回路１９に送られるような制御を行う。また、これら重要な部分以外の映像のデータに対しては、バッファメモリ１４の内容量が適正値を越えた場合に破棄することで当該バッファメモリ１４の内容量の適正化を図っている。
【０１１１】
これによっても、本実施例の通信システムの受信側９１では、上記バッファメモリ１４の入力から出力までの時間遅れを適正値に回復させることができ、また、上記映像の重要部分については破棄されないために、後のモニタディスプレイ２４に表示される映像は、高品質を保つことができるようになる。
【０１１２】
さらに、本発明の通信システムにおいて、送信側９０から受信側９１に送る画像の画質を選択する具体例としては、例えば図１１に示すように、映像データ圧縮回路６９における圧縮率を可変する構成とすることもできる。なお、この図１１において、前述した図１と同様の構成要素には同一の指示符号を付してその詳細な説明については省略する。
【０１１３】
すなわちこの図１１において、送信部９０では、上記受信部９１からの制御信号を、圧縮のパラメータを指定する圧縮パラメータ指定回路８５で受ける。この圧縮パラメータ指定回路８５は、上記制御信号に応じて映像データ圧縮回路６９における圧縮率を可変する。例えば、圧縮率を高くすることで受信側９１に送る映像データの量を減らすことができ、また、圧縮率を低くすることで受信側９１に送る映像データの量を増やすことができる。言い換えれば、圧縮率を高くすることは画質を落とすことであり、圧縮率を低くすることは画質を高めることである。なお、この図１１のように映像データの圧縮率を可変した場合には、その圧縮パラメータを受信側９１に送る必要があり、受信側９１の伸張Ｄ／Ａ変換回路１９ではこの圧縮パラメータに基づいて圧縮された映像データの伸張を行うことになる。
【０１１４】
また、図１１の構成では、音声データを圧縮する音声データ圧縮回路６７に対しても同様に可変した圧縮パラメータを送るようにすれば、受信側９１の音声データバッファメモリ１３の内容量の制御が可能となる。
【０１１５】
次に、本発明の他の実施例の構成を図１２に示す。
【０１１６】
この図１２において、送信側ホストコンピュータには、モニタディスプレイ４０とマイクロホン４１とビデオカメラ４２が接続され、またネットワーク４３を介して受信側ホストコンピュータと接続される。
【０１１７】
当該送信側ホストコンピュータにおいて、ＣＰＵ３０は、メインメモリ３１に保持されているプログラムデータを用いて各部を制御すると共に、動画の画質選択又はコマ落とし数を可変する可変手段及び上記可変手段を制御する制御手段としての機能を備えている。すなわち、このＣＰＵ３０は、前記図１の取り込み周期指定回路７１及びフレーム取り込みクロック発生回路７０やデータ処理回路９４と同様の機能をも有する。
【０１１８】
ビデオカメラ４２からの映像信号は、Ａ／Ｄ変換圧縮処理部３８によってＡ／Ｄ変換されると共に前述したような映像データ用の圧縮処理が施され、バッファメモリ３４に一時蓄えられる。さらに、このバッファメモリ３４は、前記動画信号を取り込む取り込み手段としても動作するものである。なお、図１の実施例では、前記取り込み手段としてのフレームバッファ６８には圧縮前の映像データが取り込まれる例を挙げているが、この図１２に示すバッファメモリ３４の例では圧縮後の映像データを取り込むようにしている。
【０１１９】
一方、マイクロホン４１からの音声信号は、Ａ／Ｄ変換圧縮処理部３７によってＡ／Ｄ変換されると共に前述したような音声データ用の圧縮処理が施され、バッファメモリ３３に一時蓄えられる。
【０１２０】
また、フレームメモリ３２には、例えばＣＰＵ３０によって形成された映像フレームデータや、ビデオカメラ４２によって撮影された映像に基づく映像フレームデータが記憶される。当該フレームメモリ３２からの映像フレームデータは、ディスプレイインタフェース３６を介してモニタディスプレイ４０に送られて表示される。
【０１２１】
さらに、上記バッファメモリ３３，３４に蓄えられた圧縮された音声データと映像データは、例えばＣＰＵ３０によってマルチプレクスされた後にバッファメモリ３５に一旦蓄えられてから読み出される。当該バッファメモリ３５から読み出されたデータは、ネットワークインタフェース３９に接続されたネットワーク４３を介して、受信側ホストコンピュータに送られる。このネットワークインタフェース３９が、前記図１のネットワーク送信部７及びネットワーク受信部７２と同様の機能を有している。
【０１２２】
次に、受信側ホストコンピュータには、モニタディスプレイ６１とスピーカ６２が接続され、さらにネットワーク４３を介して送信側ホストコンピュータと接続される。
【０１２３】
当該受信側ホストコンピュータにおいて、ＣＰＵ５２は、メインメモリ５３に保持されているプログラムデータを用いて各部を制御したり、また、各種の演算を行う。また、このＣＰＵ５２は、前記図１のバッファメモリ内容量の測定回路２０及び演算判断回路８１や検出回路２５，データ処理回路２６としての機能をも有している。
【０１２４】
上記ネットワーク４３を介して上記送信側ホストコンピュータから送られてきたデータは、ネットワークインタフェース５０を介してバッファメモリ５１に一旦蓄えられた後、読み出される。なお、このネットワークインタフェース５０は、前記図１のネットワーク受信部１１及びネットワーク送信部７３と同様の機能を有するものである。
【０１２５】
当該バッファメモリ５１から読み出された受信データからは、例えばＣＰＵ５２によって上記圧縮された音声データと映像データとが分離され、それぞれ対応するバッファメモリ５４，５５に送られる。
【０１２６】
バッファメモリ５４に送られた圧縮された映像データは、当該バッファメモリ５４から読み出されてデータ伸張処理部５６に送られて上記送信側ホストコンピュータでの圧縮に対応する伸張処理がなされる。当該伸張された映像データは、フレームメモリに蓄えられてフレームとなされ、ディスプレイインタフェース５８を介してモニタディスプレイ６１に送られて表示される。
【０１２７】
また、バッファメモリ５５に送られた圧縮された音声データは、当該メモリ５５から読み出されてデータ伸張処理部５９に送られて上記送信側ホストコンピュータでの圧縮に対応する伸張処理がなされる。当該伸張された音声データは、Ｄ／Ａ変換処理部６０によってアナログの音声信号に変換された後、スピーカ６２に送られる。
【０１２８】
上述したように、本発明の各実施例によれば、受信側は、送信側から送られてくる動画信号の取り込み時間と再生する時間との差である測定値Ｖが、上限（Ｖｍａｘ）内を越えたこと又は下限（Ｖｍｉｖ）内を下回ったことを検出すると、これら上限と下限との間にあるように画質又は単位時間内の送信コマ数を制御するための制御信号を送信側に送るようにし、受信側ではこの制御信号を受けて画質又は送信コマ数制御を行うことで、入力から出力までの時間遅れ及び時間のずれを適正な値に回復し、また映像の途切れを防ぐことを可能としている。
【０１２９】
また、本実施例の通信システムにおいては、受信側９１にデータ検出回路２５を設け、このデータ検出回路２５によって受信信号が音声又は動画のうちいずれかの信号を除くデータであることを検出したときには、データ処理回路２６においてそのまま処理するようにしたことで、受信信号が例えばテキストデータなどである場合に、データの内容が変化することを防いでいる。
【０１３０】
さらに、本実施例の通信システムにおいては、検出回路２５によって受信信号から音声，映像において特に重要な部分を検出したときには、演算判断回路８１においてこれら重要な部分を除く他の部分を音声データバッファメモリ１３及び映像データバッファメモリ１４に対して廃棄させることで、これらバッファメモリ１３及び１４の入力から出力までの時間遅れを適正値に回復させるようにし、上記特に重要な部分についてはそのまま処理することで、この部分の信号については破棄されることを防いでいる。
【０１３１】
【発明の効果】
上述のように本発明の通信システムにおいては、受信側において、動画信号の取り込み時間と再生する時間との差が第１の時間内を越えたこと及び／又は第２の時間内を下回ったことを検出すると、第１の時間と第２の時間との間に画質又は単位時間内の送信コマ数を制御するための制御信号を送信側に送るようにし、これを受けた送信側は、送信する動画信号から動画の画質選択又はコマ落とし数を可変する制御を行うようにしているため、受信側におけるバッファ手段の入力から出力までの時間遅れを適正値に回復させることができ、また、映像の途切れを防ぐことが可能となる。
【０１３２】
また、本発明の通信システムにおいては、受信側に第３の検出手段を設け、この第３の検出手段によって受信信号が動画信号を除く信号であることを検出したときにはそのまま処理することで、受信信号が例えばテキストデータやプログラムデータなどである場合に、データの内容が変化することを防止できる。
【０１３３】
さらに、本発明の通信システムにおいては、受信側にさらに第４の検出手段を設け、この第４の検出手段によって受信信号の動画信号の所定の部分を検出したときには、その所定の部分を除く他の部分を廃棄することで、バッファ手段の入力から出力までの時間遅れを適正値に回復させるようにし、その所定の部分についてはそのまま処理することで、この所定の部分の信号については破棄されることを防止できる。
【図面の簡単な説明】
【図１】本発明実施例の通信システムの送信側及び受信側の概略構成を示すブロック回路図である。
【図２】本発明通信システムが適用される情報処理通信システムの情報処理装置のモデルを示す図である。
【図３】マルチメディアデータの転送について説明するための図である。
【図４】ネットワークでつながれたホスト間のマルチメディアデータの転送について説明するための図である。
【図５】システム全体における情報処理装置の位置付けについて説明するための図である。
【図６】無圧縮の場合の送受信バッファサイズと転送遅延、効率の関係を示す特性図である。
【図７】ＪＰＥＧ圧縮の画像の送受信バッファサイズと転送遅延、効率の関係を示す特性図である。
【図８】情報処理装置と遅延との関係を説明するための図である。
【図９】本実施例通信システムの受信側における映像データバッファメモリの内容量測定とこの測定値が上限を越えた旨又は下限を下回った旨を示す制御信号の出力の処理を説明するためのフローチャートである。
【図１０】本実施例通信システムの送信側における画質選択又はコマ落とし制御の処理を説明するためのフローチャートである。
【図１１】本実施例の通信システムの送信側における画質選択を行う具体的構成を示すブロック回路図である。
【図１２】本発明の他の実施例の送信側ホストコンピュータと受信側ホストコンピュータの概略構成を示すブロック回路図である。
【図１３】従来の送信側と受信側からなる通信システムの概略構成を示すブロック回路図である。
【符号の説明】
１，４１・・・マイクロホン
２，４２・・・ビデオカメラ
５・・・マルチプレクサ
６・・・データバッファメモリ
７，７３・・・ネットワーク送信部
８・・・ネットワーク
９，７２・・・ネットワーク受信部
１２・・・分離回路
１３・・・音声データバッファメモリ
１４・・・映像データバッファメモリ
２０・・・バッファメモリ内容量測定回路
１８・・・音声データの伸張Ｄ／Ａ変換回路
１９・・・ビデオデータの伸張Ｄ／Ａ変換回路
２４，４０，６１・・・モニタディスプレイ
２５・・・検出回路
２６・・・受信側のデータ処理回路
２７，６２・・・スピーカ
３０・・・ＣＰＵ
３１，５３・・・メインメモリ
３２，５７・・・フレームメモリ
３３，３４，３５，５１，５４，５５・・・バッファメモリ
３６，５８・・・ディスプレイインタフェース
３７・・・音声データＡ／Ｄ変換圧縮部
３８・・・動画データＡ／Ｄ変換圧縮部
５６・・・音声データ用データ伸張部
５９・・・映像データ用データ伸張部
６０・・・音声データＤ／Ａ変換部
６５，６６・・・Ａ／Ｄ変換回路
６７・・・音声データ圧縮回路
６８・・・フレームバッファ
６９・・・映像データ圧縮回路
７０・・・フレーム取り込みクロック発生回路
７１・・・取り込み周期指定回路
８１・・・演算判断回路
９０・・・送信側
９１・・・受信側
９４・・・送信側のデータ処理回路[0001]
[Industrial application fields]
The present invention relates to a communication system, a receiving apparatus, and a method used in the technical field of communication such as a computer network, and more particularly, to a communication system, a receiving apparatus, and a method that can handle multimedia data such as audio and video data. Is.
[0002]
[Prior art]
In recent years, the performance of personal computers and workstations has been remarkably improved, and is becoming popular. Recently, there has been a demand to use them for daily communication like television and telephone, and a request to use these information processing systems for a wider range of applications. Therefore, in order to realize these requirements, the personal computer, workstation, and the like need to be able to handle multimedia data such as voice and moving images interactively in a distributed environment.
[0003]
Here, as an information processing communication system used in conventional multimedia communication, the one shown in FIG. 13 is known.
[0004]
In FIG. 13, for example, audio data, moving image data, and the like are supplied from the input devices 100 and 101. Examples of the input devices 100 and 101 include an A / D converter that converts an analog audio signal from a microphone or an analog video signal from a video camera into a digital signal. Data from the input devices 100 and 101 are sent to the input data processing devices 102 and 103, respectively, and subjected to predetermined data processing. An example of data processing in the input data processing devices 102 and 103 is data compression processing. The compressed data from the input data processing devices 102 and 103 is multiplexed by the multiplexer 104, stored in the transmission buffer memory 105, and then read out from the network transmission unit 106 via a network such as Ethernet. Sent to the network receiving unit 111 on the receiving side.
[0005]
The data received by the network reception unit 111 is temporarily stored in the reception buffer memory 112 and then read, and further, the separation device 113 performs separation processing corresponding to the multiplexing processing on the transmission side to output data processing. Sent to devices 114 and 115. The output data processing devices 114 and 115 perform data decompression processing corresponding to the input data processing devices 102 and 103, and then send them to the playback devices 116 and 117. As the playback devices 116 and 117, a D / A converter can be cited as an example.
[0006]
[Problems to be solved by the invention]
Incidentally, the above-described conventional information processing communication system has the following problems.
[0007]
First, in the conventional information processing communication system, for example, processing may be delayed due to load fluctuations in the output data processing devices 114 and 115 on the receiving side, and the content in the reception buffer memory 112 may increase excessively. As described above, when the contents of the reception buffer memory 112 increase too much, the time delay from the input to the output of data in the reception buffer memory 112 increases.
[0008]
In the example of FIG. 13, the reception buffer memory is singular. However, when there are a plurality of reception buffer memories, for example, the difference in content between the plurality of reception buffer memories is too large. In this case, the time lag between the buffer memories at the time of output (that is, time lag between media) becomes large even though the plurality of reception buffer memories must be synchronized.
[0009]
This conventional defect will be described more specifically. For example, multimedia data such as moving images and audio have properties that are essentially different from numerical data and text data that have been processed by computers. That is, multimedia data is
First, multimedia data is not just a sequence of bytes, it has a time attribute either explicitly or implicitly,
Second, the amount of data is inherently huge and redundant, and even if the hardware processing capability is improved, it is necessary to compress the data to handle it efficiently.
Third, interactive processing is required, and performance such as response and delay as well as throughput is important.
It has such properties.
[0010]
Therefore, in order to handle these in a distributed environment, it is necessary to introduce a new method for inter-media synchronization, communication, processing, resource management, and the like.
[0011]
Therefore, the present invention has been made in view of the above-described circumstances, and provides a communication system, a receiving apparatus, and a method in which a time delay from input to output is small and a time lag between media at the output is small. It is intended to do.
[0012]
[Means for Solving the Problems]
The present invention has been proposed in order to achieve the above-described object, in which multimedia data including a moving image signal is packetized and transmitted from the transmission side to the reception side, and the time taken in by the transmission side and the time reproduced by the reception side The transmission side is a real-time communication system in which the difference is within a predetermined time, and the transmission side selects the image quality for the input video signal or changes the number of dropped frames, and through this variable means Transmitting means for transmitting the packet of the moving image signal obtained to the receiving side, control means for controlling the variable means, and control signal input means for inputting a control signal from the receiving side to the control means. The receiving side captures the moving image signal transmitted from the transmitting side; and buffer means for buffering the captured moving image signal by writing it into a memory; The difference between the reproduction means for reproducing the buffered video signal and the video signal capture time and the reproduction time based on the time delay from input to output obtained from the difference between the read address and write address of the buffer means The difference between the first detection means for detecting that the time is within the first time as the upper limit and the time delay from the input to the output in the buffer means to the output of the video signal is the lower limit. The second detection means for detecting that the time is within the second time, and the difference between the video signal capture time and the playback time exceeds the first time and / or the second time Is detected, the difference between the signal capture time and the playback time based on the time delay from the input to the output in the buffer means is the second time. Those characterized by comprising a control signal transmitting means for sending a control signal for controlling the number of transmission frames in the image quality or the unit time relative to the transmission side in order to make a between the time.
[0013]
Further, the receiving side of the communication system of the present invention is provided with third detecting means for detecting that the received signal is a signal excluding the moving image signal, and the receiving side also receives the signal excluding the moving image signal, and When the third detection means detects that the received signal is a signal excluding the moving image signal, the signal excluding the moving image signal is processed as it is.
[0014]
Further, the receiving side is provided with a fourth detecting means for detecting that the predetermined portion of the moving image signal is detected. On the receiving side, when the predetermined portion of the moving image signal is detected by the fourth detecting means, Only a predetermined part is processed as it is, and the other part is discarded.
[0015]
[Action]
According to the communication system of the present invention, when the receiving side detects that the difference between the signal capture time and the playback time exceeds the first time and / or falls below the second time, A control signal for controlling the image quality or the number of frames to be transmitted within a unit time for the transmission side in order to make the difference between the signal capture time and the reproduction time between the first time and the second time. To send. Upon receiving this, the transmission side performs control for selecting the image quality of the moving image or changing the number of dropped frames for the moving image signal to be transmitted.
[0016]
That is, according to the communication system of the present invention, if the difference between the signal capture time and the reproduction time exceeds the upper limit of the first time on the receiving side, there is a time delay from input to output in the buffer means. Therefore, when the difference between the signal capture time and the playback time exceeds the first time on the receiving side, the image quality of the moving image to be sent on the transmitting side is reduced or frame dropping is performed. The time delay from the input to the output of the buffer means on the receiving side is recovered to an appropriate value.
[0017]
Furthermore, according to the communication system of the present invention, if the difference between the signal capture time and the playback time is less than the second time on the receiving side, the video may be interrupted. When the difference between the signal capture time and the playback time is less than the second time, receive by increasing the image quality of the video to be sent on the transmission side or eliminating (or reducing) the number of dropped frames. The video is prevented from being interrupted on the side.
[0018]
Further, according to the communication system of the present invention, the third detecting means is provided on the receiving side, and when the third detecting means detects that the received signal is a signal excluding the moving image signal, the processing is performed as it is. When the received signal is, for example, text data, program data, numerical data, or other binary data, the contents of these data are prevented from changing.
[0019]
Further, according to the communication system of the present invention, the receiving side is further provided with fourth detecting means, and when the predetermined portion of the moving image signal of the received signal is detected by the fourth detecting means, the predetermined portion is excluded. By discarding the other part, the time delay from the input to the output of the buffer means is restored to an appropriate value, and the predetermined part is processed as it is, and the signal of the predetermined part is not discarded. Like that.
[0020]
【Example】
Hereinafter, embodiments of a communication system, a receiving apparatus, and a method according to the present invention will be described in detail with reference to the drawings.
[0021]
FIG. 1 shows the configuration of a communication system according to an embodiment of the present invention.
As shown in FIG. 1, the communication system of the present embodiment has at least the reception side so that the difference between the time for capturing the moving image signal on the transmission side 90 and the time for reproduction on the reception side 91 is within a predetermined time. For example, it is a real-time communication system that controls the difference between the write time and the read time for the buffer memory within a predetermined time range, and the transmission side 90 has an A / D conversion circuit 66 as a capturing means for capturing a moving image signal. And a frame buffer 68, a frame capture clock generation circuit 70 as a variable means for selecting the image quality of a moving image or changing the number of dropped frames in the frame buffer 68, and capture as control means for controlling the frame capture clock generation circuit 70. A control signal from the receiving side 91 is input to the cycle designating circuit 71 and the capture cycle designating circuit 71. It is made and a network receiving section 72 as a control signal input means to be.
[0022]
In addition, the receiving side 91 of the communication system of the present embodiment includes a network receiving unit 11 serving as a capturing unit that captures a moving image signal transmitted from the transmitting side 90, a separation circuit 12 serving as a reproducing unit that reproduces the moving image signal, and an expansion D. / A conversion circuit 19 and first detection means for detecting that the difference (measurement value V described later) between the moving image signal capture time and the reproduction time is within a first time of an upper limit (described later Vmax). And measuring the capacity of the buffer memory as a second detection means for detecting that the difference (measurement value V) between the video signal capture time and the reproduction time is within a second time of a lower limit (Vmin described later). There is a difference between the circuit 20, the video data buffer memory 14 which is a buffer means for buffering the separated video data (moving image data), and the time of signal capture and playback. When it is detected that the first time has been exceeded (V> Vmax) or less than the second time (V <Vmin), the difference between the signal capture time and the reproduction time is determined as the first time. The calculation judgment as a control signal transmission means for sending a control signal for selecting the image quality or controlling the number of transmission frames within the unit time to the transmission side 90 so as to be between the second time and the second time. A circuit 81 and a network transmission unit 73 are provided.
[0023]
That is, in the communication system according to the present embodiment, when the transmission side 90 receives the control signal from the reception side 91, the moving image to be sent to the reception side 91 by changing the image quality selection or the number of dropped frames in the frame buffer 68. The amount of data is controlled.
[0024]
In the communication system according to the present embodiment, audio data is handled together with moving image data.
[0025]
In addition, the communication system according to the embodiment of the present invention can also receive signals excluding audio and video signals. Therefore, the transmission side 90 is provided with a data processing circuit 94 for processing signals excluding audio and video signals. At the same time, the receiving side 91 is further provided with a detection circuit 25 as third detection means for detecting that the received signal is a signal excluding audio and video signals. Examples of signals excluding the audio and moving image signals include text data, program data, numerical data, and other binary data. When the receiving circuit 91 detects that the received signal is a signal excluding a moving image signal in the detection circuit 25, only the text data and other parts are separated by the separation circuit 12 in accordance with the detection output from the detection circuit 25. The data processing circuit 26 processes the separated text data and the like as they are.
[0026]
Further, the detection circuit 25 on the reception side 91 of the communication system according to the embodiment of the present invention also operates as a fourth detection unit that detects, for example, an important part of video or audio as a predetermined part from the received signal. The detection output from the detection circuit 25 as the detection means 4 is sent to the arithmetic determination circuit 81. In the arithmetic determination circuit 81, the audio data buffer memory 13 and the video data buffer memory 14 are subjected to read control so that only the predetermined portion is processed as it is and the other portions are discarded.
[0027]
Here, first, prior to specific description of the communication system of the embodiment of the present invention shown in FIGS. 1 and 9 to 12, information corresponding to multimedia data of the information processing communication system to which the communication system of the present invention is applied. Basic functions of processing device, model of information processing device, so-called application programming interface (API), positioning of information processing device corresponding to multimedia in information processing communication system, evaluation of information processing Will be described in the following items.
[0028]
1. Functions of information processing device for multimedia in information processing communication system
The goal realized by this information processing device is
(1) Although there are various restrictions, it must be built on a standard personal computer or workstation to maintain compatibility with existing systems.
(2) Audio and video communication and special processing are left to the information processing device, and the client program only performs its control.
(3) The ability to deal flexibly according to the attributes of the media data and the purpose of the client program,
(4) Capability to respond to changes in the load of resources such as processors and networks
(5) The model is simple,
Etc.
[0029]
Due to these requirements, the information processing apparatus needs to have the functions described below.
[0030]
1.1 Media type
As media data handled by the information processing apparatus, there are basically audio and moving images. These are assigned temporal attributes such as the amount of data per unit time and generation time, and are used for processing in the information processing apparatus. For example, in the case of audio data with a data width of 8 bits and a sampling frequency of 8 kHz, the attributes of unit time is 1/8000 (sec) and the data amount per unit time is 1 byte.
[0031]
1.2 Information processing device and media device
The information processing apparatus outputs data from a plurality of input devices to the output device while maintaining temporal synchronization.
[0032]
As an input device,
(1) Hardware devices such as audio interface and video input interface,
(2) Multimedia data files such as sound files, movie files and movie files,
(3) Multicast address,
(4) Client process
And so on.
[0033]
As an output device,
(1) Devices such as audio interfaces and windows,
(2) multimedia files,
(3) Multicast address,
(4) Client process,
And so on.
[0034]
There is only one information processing apparatus in a host, and the device that it directly handles must be on that host. When the input device and the output device need to be on different hosts, the client accesses each information processing apparatus on each host and connects the information processing apparatuses.
[0035]
1.3 Media data transfer and synchronization
The following can be considered as criteria for evaluating the quality of an object provided to a user as a multimedia system.
For example,
(1) Tolerance limit of transfer delay,
(2) Tolerance limit for synchronization between media,
(3) throughput,
(4) Data loss is allowed and not allowed
It is.
[0036]
In order to obtain satisfactory quality on a given transfer path, various parameters at the time of transfer such as data amount, compression method, protocol and packet size are controlled. The information processing apparatus performs synchronization between media in accordance with the quality evaluation standard.
[0037]
1.4 Network protocol
The multimedia data network protocol used by the information processing apparatus needs to be able to dynamically control the flow and the like in consideration of quality evaluation criteria. 100Mbit / sec token passing fiber using so-called Ethernet based on IEEE 802.3 standard of current network environment and optical fiber proposed in SC13 of International Organization for Standardization (ISO) Since network devices such as a distributed data interface (FDDI) share and share resources individually in a distributed environment, services that secure network resources in advance are difficult. However, considering the compatibility with the current network environment, it is necessary to use the Internet protocol (IP), and this time, as a network protocol, the so-called transmission control protocol (TCP) and connection A so-called user datagram protocol (UDP), which is a protocol of a lesser form, was used.
[0038]
1.5 Data compression / decompression
The information processing apparatus has a function of compressing / decompressing audio data and moving image data by software or hardware. As a speech compression method, the recommendation of G. ITU standard for speech coding standards of the International Telegraph and Telephone Consultative Committee (CCITT). 711, G. 721, G. 722, G.G. Supports standards such as 728. In addition, as an image compression system, the international standardization working group of the color still image coding system of the International Telegraph and Telephone Consultative Committee (CCITT), so-called JPEG (Joint Picture Expert Group), and a video coding recommendation for a video conference system. H. 261, there is a system such as MPEG (Moving Picture Expert Group) of the international standardization working group of the color moving picture coding system. Since each compression method has its characteristics, it is necessary to use it properly depending on the application.
[0039]
1.6 Resource control
The information processing apparatus has a function of processing input / output requests from a plurality of client programs. For example, when there is a voice input request from a plurality of clients even though there is only one voice input device, the following processing method can be considered.
(1) Copy and distribute to all request sources.
(2) Prioritize first-come-first-served requests and reject later-arrival requests.
(3) Switch clients sequentially. This function is used by a program such as a multimedia manager corresponding to the window manager in the window system.
[0040]
1.7 Physical device control
In order to handle multimedia data such as audio and moving images, it is necessary to connect a so-called audio / visual device (AV device) such as a video camera or a video deck. The information processing apparatus supports a predetermined protocol for AV device control for these controls.
[0041]
2. Information processing device model and API
2.1 Model of information processing equipment to which AV equipment is connected
As illustrated in FIG. 2, the information processing apparatus generates one execution control unit (AVobj) for each client. When the client wants to input / output multimedia data, it makes a request to the information processing apparatus in the following procedure. First, a virtual media device (AVdev) is opened in the execution control unit (AVobj). Since the virtual media device (AVdev) is not a physical device, exclusive control or opening from multiple devices can be realized.
[0042]
In addition, as shown in FIG. 3, the execution control unit (AVobj) that opened the virtual media device (AVdev) for input and the execution control unit (AVobj) that opened the media device (AVdev) for output. By connecting, a transfer path for multimedia data is secured. Media synchronization is guaranteed between devices opened in the same execution control unit (AVobj).
[0043]
Further, as shown in FIG. 4, an execution control unit (AVobj) is generated and connected in information processing apparatuses on different hosts connected by a network, and multimedia data is transferred at a workstation in a distributed environment.
[0044]
A telephone can be realized by using a sound device as a device (AVdev) opened by the execution control unit (AVobj) on the sending and receiving sides. Furthermore, videophone can be realized by opening a video device. When a movie file is specified as an input device and a sound device and a video device are opened as an output device, a movie player is obtained. In this way, various multimedia applications can be easily created by rearranging input / output devices.
[0045]
2.2 API of information processing equipment
For example, the following library is prepared in the library of the information processing apparatus.
[0046]
int avs _new (char * hostname);
This generates an execution control unit (AVobj) in the information processing apparatus activated on the host name (hostname). If an error occurs, null is returned. If the process ends normally, the ID (identification information) of the execution control unit (AVobj) is returned. All instructions for the execution control unit (AVobj) are executed using the ID of this execution control unit (AVobj).
[0047]
int avs _open (int net, char * devname, int mode);
This opens the device (AVdev). The arguments are the execution control unit (AVobj) ID, device name, and mode.
[0048]
int avs _connect (int net1, int net2);
This is a point-to-point connection between two execution control units (AVobj). As a result, when one of the connected execution control units (AVobj) enters the transfer state by the function (avs_transfer) described below, the other execution control unit (AVobj) can receive the data and automatically receives it. Process the data. Since a plurality of reception execution control units (AVobj) can be connected to one transmission execution control unit (AVobj), one-to-many data transfer can be processed.
[0049]
int avs _transfer (int net, int dev, int length);
This controls the transmission of execution control units (AVobj). If 0 is specified as the device ID, the execution control unit (AVobj) is valid for all open devices. If a positive number is designated for the length (length), data transfer is performed for that length (unit: msec). If zero is specified here, transfer is performed until the next function (av_transfer) is given. If a negative number is specified, stop immediately.
[0050]
int avs _destroy (int net);
This releases the execution control unit (AVobj) generated by the function (av_new).
[0051]
int avs _interval (int net, int dev, int interval);
This sets the transfer interval. The unit is msec. If zero is set as the device ID, it is applied to all devices opened in the execution control unit (AVobj).
[0052]
int avs _resize (int net, int dev, int width, int height);
This requires a resize to the video device. The unit of size is pixels. If zero is set as the device ID, it is applied to all video devices opened in the execution control unit (AVobj).
[0053]
int avs _nettype (int net, char * type);
This sets the type of network that connects execution control units (AVobj). Currently, the TCP and UDP are supported. This must be done before executing the function (avs_connect).
[0054]
int avs _fd (int net);
This returns the file descriptor of the control connection connection between the client and the execution control unit (AVobj).
[0055]
int avs _codec (int net, char * type, int quality);
This specifies the compression method of media data. Here, only JPEG compression / decompression is supported for image data.
[0056]
In addition, there are the following libraries for accessing media data directly to the device (AVdev).
For example,
int avs _read (int net, int dev, int shmid, int size);
int avs _write (int net, int dev, int shmid, int size);
int avs _ioctl (int net, int dev, int request, int shmi);
It is.
[0057]
3. Implementation of information processing equipment
The positioning of the information processing apparatus in the entire system is as shown in FIG.
3.1 Use of Thread
When threads are used, the time required for context switching is shorter than that achieved by using a plurality of processes, and an environment such as a memory can be shared among the threads, so that programming becomes easy. In realizing the information processing apparatus, there are two possible methods for assigning threads to the model.
[0058]
For example,
Assign to a thread for each function,
Assign a thread for each data stream,
There is.
The former is a method in which a thread is assigned for each function such as screen input / output, audio input / output, and network, and a pipeline is formed. The latter is a method in which a thread is assigned to processing from input to output for each media data.
[0059]
Even if a pipeline is formed for each function, there is no advantage unless it is in a multiprocessor environment, and the latter is suitable for schedule / priority control in units of streams in multiple streams. We also used a method of assigning threads to.
[0060]
3.2 TCP / IP send / receive buffers and delays
In order to realize high reliability in the TCP / IP, packet ordering, checksum, timeout, and retransfer are performed, and the overhead is large and data transfer delay becomes a problem.
[0061]
Therefore, a buffer is prepared here for transmission and reception. For example, in a workstation, the default is 8K (bytes). A delay is caused by the accumulation of data in this buffer. For example, if the screen has a resolution of 160 × 120 and a depth of 16 bits, one screen is approximately 38 Kbytes, and even if the buffers on both the transmission side and the reception side are combined, one frame is not buffered. In this case, increasing the buffer size improves the transfer efficiency. However, if image compression is applied to a size of about 4 Kbytes, which is about 1/10, about 4 frames are accumulated in both buffers. If transfer is performed at a speed of 5 frames per second, this alone is a delay of about 1 second.
[0062]
However, if the buffer is made too small, the transfer efficiency deteriorates, so there is a trade-off here. The size of the transmission / reception buffer can be changed by a function (setsockopt).
[0063]
4). Evaluation of information processing equipment
Evaluation when the information processing apparatus is mounted on a workstation is as follows.
[0064]
4.1 Transmission / reception buffer and transfer delay, efficiency
As an example of actually measuring the size, transfer delay, and efficiency of the transmission / reception buffer, moving image data was transferred between two workstations connected to the same network. Here, the workstation is connected via Ethernet, the resolution is 160 x 120, 16-bit deep uncompressed and JPEG compressed images are measured, and the size of one frame is uncompressed, approximately 38 Kbytes, JPEG compressed About 5K bytes.
[0065]
The transfer delay was measured using the function (timed (8)), and the time from image capture to transfer and display on the remote side was measured. Further, the maximum transfer frame rate is measured as the transfer efficiency. FIG. 6 shows the relationship between the transmission / reception buffer size, transfer delay, and efficiency of an uncompressed image and FIG.
[0066]
As shown in FIG. 6, when the size of one frame is large without compression, since one frame does not fit in the transmission / reception buffer, the difference in transfer delay with respect to the buffer size hardly changes. When the buffer size is 8 Kbytes, good results in both transfer delay and efficiency appear. According to FIG. 6, it can be understood that the TCP / IP transfer is most efficient in that case because the default transmission / reception buffer size is 8 Kbytes.
[0067]
In contrast, when transferring an image subjected to JPEG compression shown in FIG. 7, the maximum frame rate is small because JPEG compression / decompression is performed by software. In addition, since several frames are accumulated in the transmission / reception buffer, the delay increases as the buffer size increases. Therefore, when the image data is sufficiently smaller than the network bandwidth by performing compression such as JPEG, the transfer delay of the media data can be shortened by reducing the transmission / reception buffer size so that the transfer efficiency does not decrease.
[0068]
4.2 Screen transfer speed
In the same environment, the screen size, frame rate, and transfer delay are measured. First, when Ethernet was used as the network, the results in the following table were obtained. Table 1 shows the maximum transfer rate and delay performance of a 16-bit deep uncompressed image, and Table 2 shows a JPEG compressed image.
[0069]
[Table 1]

[0070]
[Table 2]

[0071]
Next, Table 3 shows the performance when 1B (64K) of an integrated service digitial network (ISDN) is used as the network. When ISDM is used, if the image data is not compressed, the frame rate becomes 0.2 (fps) even at the minimum screen size (80 × 60), which is impractical, and is therefore uncompressed with a depth of 16 bits. Evaluation of image transfer performance was omitted.
[0072]
[Table 3]

[0073]
By using such an information processing apparatus, it has become possible to easily create a multimedia application without being conscious of capturing and compressing / decompressing images and sounds, and network programming.
[0074]
4.4 Realization of synchronization between media
In this information processing apparatus, media data is synchronized and transferred in a single stream. Further, since TCP / IP is used as the network protocol, the order of the packets is guaranteed. Furthermore, if a plurality of media data can be fetched simultaneously and interleaved to ensure synchronization at the time of data transmission, it can be assumed that a plurality of media data are synchronized even on the data receiving side. In fact, the synchronization of audio and video media data transferred over a single network using this method has been satisfactory.
[0075]
4.5 Relation to delay
In the above implementation, standard TCP / IP is used as the network protocol in the UNIX of the operating system. Here, as shown in FIG. 8, in an environment in which media data can be sent directly, the information processing apparatus can control the buffers on the transmission side and the reception side, and fine flow rate control can be performed.
[0076]
Hereinafter, referring back to FIG. 1, a communication system according to an embodiment of the present invention that specifically realizes the above will be described.
[0077]
In FIG. 1, audio signals and video signals (moving image signals) from the microphone 1 and the video camera 2 are sent to the corresponding A / D conversion circuits 65 and 66, respectively, where they are A / D converted. The audio data from the A / D conversion circuit 65 is sent to the audio data compression circuit 67. As described above, for example, G. 711, G. 721, G. 722, G.G. Compression using any one of 728 and the like is performed. The video data from the A / D conversion circuit 66 is sent to the video data compression circuit 69 via a frame buffer 68 described later. As described above, for example, JPEG, H.P. Compression using any one of H.261, MPEG and the like is performed.
[0078]
The compressed audio data and video data from the compression circuits 67 and 69 are multiplexed by the multiplexer 5 and then sent to the network transmission unit 7 via the data buffer memory (for example, FIFO memory) 6, and the network transmission unit 7 to the network receiving unit 11 on the receiving side 91 via the network 8 such as Ethernet.
[0079]
In the communication system of FIG. 1, for example, text data, program data, numerical data, other binary data, and the like are supplied to the input terminal 89 as signals excluding the voice and moving image. These data are subjected to processing such as addition of an error correction code in the data processing circuit 94 and then output from the network transmission unit 7 via the multiplexer 5.
[0080]
The data received by the network receiving unit 11 on the receiving side 91 is separated into the compressed audio data and the video data by the separation circuit 12, and the corresponding audio data buffer memory (FIFO memory) 13 and the video data buffer memory, respectively. (FIFO memory) 14 is sent.
[0081]
The buffer memory capacity measurement circuit 20 measures the internal capacity of the memory 14 from the write address and the read address of the video data buffer memory 14, and sends the measurement result (measured value V described later) to the arithmetic determination circuit 81. send. When it is detected from the result of measuring the internal capacity of the memory 14 from the measurement circuit 20 that the arithmetic determination circuit 81 detects that the internal capacity of the video data buffer memory 14 exceeds the upper limit (Vmax described later). Is sent to the transmission side 91 via the network transmission unit 73 as a control signal.
[0082]
Further, in the calculation determination circuit 81, the internal capacity of the video data buffer memory 14 is less than the lower limit (Vmin, which will be described later) from the internal capacity measurement result (measured value V, which will be described later) of the memory 14 from the measurement circuit 20. If detected, the video may be interrupted, and a signal indicating that (below the lower limit) is sent to the transmission side 90 via the network transmission unit 73 as a control signal.
[0083]
Here, the buffer memory internal capacity measuring circuit 20 and the operation determining circuit 81 measure the internal capacity of the video data buffer memory 14 and generate the control signal according to the flowchart shown in FIG. .
[0084]
In FIG. 9, in step S11, a measured value V (unit: playback time) of the internal capacity of the video data buffer memory 14 is obtained. The measured value V is obtained by the following formula.
V = (buffer read address) − (buffer write address)
[0085]
In the next step S2, V> Vmax is determined from the measured value V and the upper limit value Vmax of the appropriate range of the internal capacity of the video data buffer memory 14, and if it is determined NO, the process proceeds to step S32, and conversely. If it is determined as YES, the process proceeds to step S35.
[0086]
In step S35, a signal indicating that V> Vmax is sent as a control signal to the transmission side 90, and then the process returns to step S11.
[0087]
If it is determined NO in step S12 and the process proceeds to step S32, V <Vmin is determined in step S32 from the measured value V and the lower limit value Vmin of the appropriate range of the internal capacity of the video data buffer memory 14. To do. If it is determined as NO in step S32, the process returns to step S11. If YES is determined, the process proceeds to step S36.
[0088]
In step S36, a signal indicating that V <Vmin is transmitted to the transmission side 90 as a control signal, and the process returns to step S11.
[0089]
If the appropriate value of the appropriate range of the internal capacity of the video data buffer memory 14 is Vsta, the relationship between the upper limit value Vmax, the appropriate value Vsta, and the lower limit value Vmin is Vmin <Vsta <Vmax.
[0090]
The control signal from the receiving side 91 is received by the network receiving unit 72 of the transmitting side 90 and is sent to the capture cycle specifying circuit 71. The capture cycle designation circuit 71 controls the frame capture clock generation circuit 70 based on the control signal. When the frame capture clock generation circuit 70 receives the signal from the capture cycle designating circuit 71, the frame capture clock generation circuit 70 generates a frame capture clock that varies the number of dropped frames in the frame buffer 68. Here, for example, by dropping frames (or increasing the number of dropped frames), the amount of video data to be transmitted can be reduced, and video data to be transmitted without dropping frames or by reducing the number of dropped frames The amount of can be increased.
[0091]
That is, the transmission side 90 performs the processing of the flowchart as shown in FIG. 10 according to the control signal from the reception side 91.
[0092]
In FIG. 10, in step S41, a control signal from the receiving side 91 is received.
[0093]
In the next step S42, it is determined whether or not the received control signal is a signal representing V> Vmax. If it is determined that the signal is a signal indicating V> Vmax, the process proceeds to step S43. Then, the process proceeds to step S44.
[0094]
In step S43, the capture cycle designation circuit 71 controls the frame capture clock generation circuit 70 so that the frame capture clock cycle in the frame buffer 68 is increased by P. After step S43, the process returns to step S41. Note that the value of P is, for example, 1%.
[0095]
On the other hand, in step S44 when it is determined NO in step S42, it is determined whether or not the received control signal is a signal representing V <Vmin, and when it is determined that the signal represents V <Vmin. Advances to step S45, and returns to step S41 if determined NO.
[0096]
In step S44, the capture cycle designation circuit 71 controls the frame capture clock generation circuit 70 to shorten the frame capture clock cycle in the frame buffer 68 by P. After step S44, the process returns to step S41. Note that the value of P is, for example, 1%.
[0097]
As a result, in the communication system of the present embodiment, the content of the video data buffer memory 14 on the receiving side 91 increases, and the time delay and time shift from input to output in the video data buffer memory 14 are appropriate values. Even if it exceeds the time limit, the time feed and time lag can be recovered to appropriate values, and even if the content of the video data buffer memory 14 is reduced and the video may be interrupted, The interruption of the video can be prevented.
[0098]
On the receiving side 91, the audio data and the video data read from the audio data buffer memory 13 and the video data buffer memory 14 are respectively subjected to corresponding expansion processing and D / A conversion expansion D / A conversion circuit 18, 19 is sent. In the decompression D / A conversion circuits 18 and 19, decompression processing corresponding to each compression processing in the A / D conversion compression circuits 3 and 4 is respectively performed, and then D / A conversion is performed and output.
[0099]
The audio signal from the expansion D / A conversion circuit 18 is sent to the speaker 27, and the video signal from the expansion D / A conversion circuit 19 is sent to the monitor display 24.
[0100]
Further, in the configuration of FIG. 1, by changing the number of dropped frames on the transmission side 90, the time delay from the input to the output and the time lag in the video data buffer memory 14 can be recovered to appropriate values. However, the same thing can be done by selecting the image quality of the image sent from the transmission side 90 to the reception side 91, for example.
[0101]
That is, when this image quality is selected, for example, the image data is secured by reducing the amount of data to be transmitted by thinning out each pixel data captured in the frame buffer 68, or conversely not performing the thinning. In this case, on the receiving side 91, when the pixel data is thinned out, for example, processing such as interpolation is performed to restore the image.
[0102]
In the communication system of FIG. 1, the received signal supplied to the separation circuit 12 on the reception side 91 is also sent to the detection circuit 25 via the separation circuit 12. The detection circuit 25 detects signals such as text data, program data, numerical data, and other binary data as signals excluding audio and moving image signals from the received signal.
[0103]
When the detection circuit 25 detects data excluding the audio and moving image signals, the detection circuit 25 outputs a detection signal to the separation circuit 12. When the detection signal from the detection circuit 25 is supplied, the separation circuit 12 separates only the data corresponding to the detection signal from the reception signal and sends it to the data processing circuit 26.
[0104]
The data processing circuit 26 performs a predetermined process on the supplied data and then outputs it from the output terminal. Examples of the predetermined process include an error correction process.
[0105]
As described above, the receiving side 91 of the present embodiment has the detection circuit 25, and the separation circuit 12 separates data excluding audio and moving images from the reception signal according to the detection signal from the detection circuit 25. By doing so, it is possible to perform processing according to data excluding these sounds and moving images.
[0106]
Also, the audio and video data separated by the separation circuit 12 on the receiving side 91 of this embodiment is also sent to the detection circuit 25.
[0107]
At this time, the detection circuit 25 detects a low-frequency part from the audio data in consideration of, for example, human auditory characteristics, or a particularly important audio part such as a human voice part. Is detected. For video data, for example, in consideration of human visual characteristics, a particularly visually important video portion is detected from the video data, or a particularly important portion is detected from video content. From the detection circuit 25, detection signals corresponding to the video data and the audio data are sent to the arithmetic determination circuit 81.
[0108]
When receiving the detection signal corresponding to the audio data from the detection circuit 25, the arithmetic determination circuit 81 performs control so that the audio portion corresponding to the detection signal is not discarded for the audio data buffer memory 13. That is, control is performed so that an important portion of the sound corresponding to the detection signal is sent from the audio data buffer memory 13 to the subsequent audio data expansion D / A conversion circuit 18 as it is. On the other hand, audio data other than these important portions is discarded when the internal capacity of the buffer memory 13 exceeds an appropriate value, for example, to optimize the internal capacity of the buffer memory 13.
[0109]
Also by this, the receiving side 91 of the communication system of the present embodiment can recover the time delay from the input to the output of the buffer memory 13 to an appropriate value, and the important part of the voice is not discarded. In addition, the sound output from the later speaker 27 can maintain high quality.
[0110]
On the other hand, when receiving the detection signal corresponding to the video data from the detection circuit 25, the arithmetic determination circuit 81 performs control so that the video portion corresponding to the detection signal is not discarded for the video data buffer memory 14. That is, control is performed so that an important part of the video corresponding to the detection signal is sent from the video data buffer memory 14 to the subsequent video data decompression D / A conversion circuit 19 as it is. Further, the video data other than these important portions is discarded when the internal capacity of the buffer memory 14 exceeds an appropriate value, so that the internal capacity of the buffer memory 14 is optimized.
[0111]
This also allows the receiving side 91 of the communication system of the present embodiment to recover the time delay from the input to the output of the buffer memory 14 to an appropriate value, and important parts of the video are not discarded. In addition, the video displayed on the subsequent monitor display 24 can maintain high quality.
[0112]
Further, in the communication system of the present invention, as a specific example of selecting the image quality of the image to be sent from the transmission side 90 to the reception side 91, for example, as shown in FIG. You can also In FIG. 11, the same components as those in FIG. 1 described above are denoted by the same reference numerals, and detailed description thereof is omitted.
[0113]
That is, in FIG. 11, in the transmission unit 90, the control signal from the reception unit 91 is received by the compression parameter designating circuit 85 that designates the compression parameter. The compression parameter designating circuit 85 varies the compression rate in the video data compression circuit 69 in accordance with the control signal. For example, the amount of video data sent to the receiving side 91 can be reduced by increasing the compression rate, and the amount of video data sent to the receiving side 91 can be increased by reducing the compression rate. In other words, increasing the compression rate is reducing the image quality, and decreasing the compression rate is increasing the image quality. When the compression rate of the video data is varied as shown in FIG. 11, it is necessary to send the compression parameter to the reception side 91, and the decompression D / A conversion circuit 19 of the reception side 91 is based on this compression parameter. Thus, the compressed video data is expanded.
[0114]
In the configuration of FIG. 11, if the variable compression parameter is similarly sent to the audio data compression circuit 67 that compresses the audio data, the internal capacity of the audio data buffer memory 13 on the receiving side 91 can be controlled. It becomes possible.
[0115]
Next, the configuration of another embodiment of the present invention is shown in FIG.
[0116]
In FIG. 12, a monitor display 40, a microphone 41, and a video camera 42 are connected to a transmission side host computer, and are connected to a reception side host computer via a network 43.
[0117]
In the transmission-side host computer, the CPU 30 controls each unit using program data stored in the main memory 31, and also controls variable means for changing the image quality selection or number of dropped frames of the moving image and the variable means. It has a function as a means. That is, the CPU 30 also has the same functions as the capture cycle designation circuit 71, the frame capture clock generation circuit 70, and the data processing circuit 94 in FIG.
[0118]
The video signal from the video camera 42 is A / D converted by the A / D conversion compression processing unit 38 and subjected to the above-described compression processing for video data, and is temporarily stored in the buffer memory 34. Further, the buffer memory 34 also operates as a capturing means for capturing the moving image signal. In the embodiment of FIG. 1, an example in which video data before compression is taken into the frame buffer 68 serving as the taking-in means is shown. However, in the example of the buffer memory 34 shown in FIG. To capture.
[0119]
On the other hand, the audio signal from the microphone 41 is A / D converted by the A / D conversion compression processing unit 37 and subjected to the compression processing for the audio data as described above, and is temporarily stored in the buffer memory 33.
[0120]
The frame memory 32 stores, for example, video frame data formed by the CPU 30 and video frame data based on video shot by the video camera 42. The video frame data from the frame memory 32 is sent to the monitor display 40 via the display interface 36 and displayed.
[0121]
Further, the compressed audio data and video data stored in the buffer memories 33 and 34 are multiplexed by, for example, the CPU 30 and then temporarily stored in the buffer memory 35 and then read out. Data read from the buffer memory 35 is sent to the receiving host computer via the network 43 connected to the network interface 39. The network interface 39 has the same functions as the network transmission unit 7 and the network reception unit 72 shown in FIG.
[0122]
Next, a monitor display 61 and a speaker 62 are connected to the reception side host computer, and further connected to the transmission side host computer via the network 43.
[0123]
In the reception-side host computer, the CPU 52 controls each unit using program data stored in the main memory 53 and performs various calculations. The CPU 52 also has functions as the buffer memory internal capacity measurement circuit 20, the operation determination circuit 81, the detection circuit 25, and the data processing circuit 26 in FIG. 1.
[0124]
Data sent from the sending host computer via the network 43 is temporarily stored in the buffer memory 51 via the network interface 50 and then read. The network interface 50 has the same functions as the network receiver 11 and the network transmitter 73 shown in FIG.
[0125]
From the received data read from the buffer memory 51, for example, the CPU 52 separates the compressed audio data and video data and sends them to the corresponding buffer memories 54 and 55, respectively.
[0126]
The compressed video data sent to the buffer memory 54 is read from the buffer memory 54 and sent to the data decompression processing unit 56 where decompression processing corresponding to the compression at the transmission side host computer is performed. The decompressed video data is stored in the frame memory to form a frame, which is sent to the monitor display 61 via the display interface 58 and displayed.
[0127]
The compressed audio data sent to the buffer memory 55 is read from the memory 55 and sent to the data decompression processing unit 59, where decompression processing corresponding to the compression at the transmission side host computer is performed. The expanded audio data is converted into an analog audio signal by the D / A conversion processing unit 60 and then sent to the speaker 62.
[0128]
As described above, according to each embodiment of the present invention, the receiving side has a measured value V that is the difference between the capture time of the moving image signal sent from the transmitting side and the playback time within the upper limit (Vmax). When it is detected that the value exceeds the lower limit (Vmiv) or falls below the lower limit (Vmiv), a control signal for controlling the image quality or the number of transmission frames within the unit time is sent to the transmission side so as to be between these upper limit and lower limit The receiving side receives this control signal and controls the image quality or the number of frames to be transmitted, thereby recovering the time delay and time lag from input to output to appropriate values, and preventing video interruptions. It is possible.
[0129]
In the communication system of the present embodiment, the data detecting circuit 25 is provided on the receiving side 91, and when the data detecting circuit 25 detects that the received signal is data excluding one of audio and video signals. Since the data processing circuit 26 performs the processing as it is, the content of the data is prevented from changing when the received signal is, for example, text data.
[0130]
Further, in the communication system of the present embodiment, when the detection circuit 25 detects a particularly important part in the audio and video from the received signal, the operation determination circuit 81 removes the other parts excluding these important parts from the audio data buffer memory. 13 and the video data buffer memory 14 are discarded so that the time delay from the input to the output of these buffer memories 13 and 14 is restored to an appropriate value, and the above particularly important parts are processed as they are. The signal of this part is prevented from being discarded.
[0131]
【The invention's effect】
As described above, in the communication system of the present invention, on the receiving side, the difference between the video signal capture time and the playback time exceeds the first time and / or falls below the second time. Is detected, the control signal for controlling the image quality or the number of frames to be transmitted within the unit time is transmitted to the transmission side between the first time and the second time, and the transmission side having received this transmits the transmission signal. Since the video quality is selected from the video signal to be controlled or the control for changing the number of dropped frames is performed, the time delay from the input to the output of the buffer means on the receiving side can be recovered to an appropriate value. Can be prevented.
[0132]
In the communication system of the present invention, the receiving side is provided with the third detecting means, and when the third detecting means detects that the received signal is a signal excluding the moving image signal, the processing is performed as it is. When the signal is, for example, text data or program data, the contents of the data can be prevented from changing.
[0133]
Further, in the communication system of the present invention, a fourth detection means is further provided on the receiving side, and when the predetermined portion of the moving image signal of the received signal is detected by the fourth detection means, the predetermined portion is excluded. By discarding this part, the time delay from the input to the output of the buffer means is restored to an appropriate value. By processing the predetermined part as it is, the signal of this predetermined part is discarded. Can be prevented.
[Brief description of the drawings]
FIG. 1 is a block circuit diagram showing a schematic configuration of a transmission side and a reception side of a communication system according to an embodiment of the present invention.
FIG. 2 is a diagram showing a model of an information processing apparatus of an information processing communication system to which the communication system of the present invention is applied.
FIG. 3 is a diagram for explaining transfer of multimedia data;
FIG. 4 is a diagram for explaining transfer of multimedia data between hosts connected by a network.
FIG. 5 is a diagram for explaining the positioning of the information processing apparatus in the entire system.
FIG. 6 is a characteristic diagram showing the relationship between transmission / reception buffer size, transfer delay, and efficiency in the case of no compression.
FIG. 7 is a characteristic diagram showing the relationship between the JPEG compressed image transmission / reception buffer size, transfer delay, and efficiency.
FIG. 8 is a diagram for explaining a relationship between an information processing apparatus and a delay.
FIG. 9 is a diagram for explaining the processing of the content of the video data buffer memory on the receiving side of the communication system of the present embodiment and the output of a control signal indicating that the measured value exceeds the upper limit or falls below the lower limit. It is a flowchart.
FIG. 10 is a flowchart for explaining image quality selection or frame dropping control processing on the transmission side of the communication system according to the embodiment.
FIG. 11 is a block circuit diagram showing a specific configuration for performing image quality selection on the transmission side of the communication system of the present embodiment.
FIG. 12 is a block circuit diagram showing a schematic configuration of a transmission side host computer and a reception side host computer according to another embodiment of the present invention.
FIG. 13 is a block circuit diagram showing a schematic configuration of a conventional communication system including a transmission side and a reception side.
[Explanation of symbols]
1,41 ... Microphone
2, 42 ... Video camera
5 ... Multiplexer
6 ... Data buffer memory
7, 73 ... Network transmitter
8 ... Network
9, 72 ... Network receiver
12... Separation circuit
13 ... Audio data buffer memory
14 ... Video data buffer memory
20 ... Buffer memory capacity measurement circuit
18 ... Decompression D / A conversion circuit for audio data
19 ... Video data expansion D / A conversion circuit
24, 40, 61 ... monitor display
25 ... Detection circuit
26: Data processing circuit on receiving side
27, 62 ... Speaker
30 ... CPU
31, 53 ... Main memory
32, 57 ... frame memory
33, 34, 35, 51, 54, 55 ... buffer memory
36,58 ... Display interface
37 ... Audio data A / D conversion compression unit
38 ... Moving image data A / D conversion compression unit
56 ... Data decompression unit for audio data
59 ... Data expansion unit for video data
60: Audio data D / A converter
65, 66... A / D conversion circuit
67. Audio data compression circuit
68 ... Frame buffer
69 ... Video data compression circuit
70... Frame capture clock generation circuit
71 ... Capture cycle designation circuit
81... Operation judgment circuit
90 ... Sending side
91 ... Receiving side
94. Data processing circuit on transmission side

Claims (5)

A real-time communication system in which multimedia data including a video signal is packetized and transmitted from a transmission side to a reception side, and a difference between a time taken on the transmission side and a reproduction time on the reception side is within a predetermined time,
The transmission side is a variable means for selecting the image quality for the input video signal or changing the number of dropped frames, and a transmission means for transmitting the video signal packet obtained through the variable means to the reception side, A control means for controlling the variable means; and a control signal input means for receiving a control signal from the receiving side to the control means,
The receiving side captures the moving image signal transmitted from the transmitting side, buffer means for buffering the captured moving image signal by writing it into the memory and reading it, and reproduces the buffered moving image signal Based on the time delay from the input to the output obtained from the difference between the read address and the write address of the reproduction means and the buffer means , the difference between the video signal capture time and the reproduction time is within the first time as the upper limit. The difference between the first detection means for detecting the presence and the time taken from the input to output of the buffer means to the output based on the time delay from the input to the output is within the second time as the lower limit. The difference between the second detection means for detecting the moving image signal capture time and the reproduction time exceeds the first time and / or When it is detected that the time is less than the second time, the difference between the signal acquisition time and the reproduction time based on the time delay from the input to the output in the buffer means is the second time. And a control signal transmitting means for sending a control signal for controlling the image quality or the number of frames to be transmitted within a unit time to the transmitting side so as to be within a predetermined time. When the control signal transmission means detects that the difference between the video signal capture time and the reproduction time exceeds the first time, the control signal transmission means reduces the transmission data amount to the transmission side. When a control signal for controlling the number of frames to be transmitted within a unit time is sent and it is detected that the difference between the video signal capture time and the playback time is less than the second time, 2. The communication system according to claim 1, wherein a control signal for controlling the image quality or the number of transmission frames within a unit time is transmitted so as to increase a transmission data amount . On the receiving side, the moving image signal transmitted from the transmitting side is input, and fourth detecting means for detecting that the moving image signal is an important part is provided.
On the receiving side, when an important part of the moving image signal is detected by the fourth detecting means, other parts other than the important part are discarded according to a time delay from input to output in the buffer means. The communication system according to claim 1 or 2. In a receiving apparatus that receives packetized multimedia data including a moving image signal transmitted from a transmitting side,
Capture means for capturing the packetized video signal transmitted from the transmission side;
Buffer means for buffering the captured video signal by writing to and reading from the memory ;
Playback means for playing back the buffered video signal;
Based on the time delay from input to output obtained from the difference between the read address and write address of the buffer means , it is detected that the difference between the video signal capture time and the playback time is within the first time as the upper limit. First detecting means for
Second detection means for detecting that the difference between the video signal capture time and the playback time is within the second time as a lower limit based on a time delay from input to output in the buffer means;
When it is detected that the difference between the capture time of the video signal and the playback time exceeds the first time and / or falls below the second time, the input from the buffer means to the output is detected. In order to set the difference between the signal capture time based on the time delay and the reproduction time to be between the first time and the second time, the image quality or the number of transmission frames within the unit time is set to the transmission side. And a control signal transmitting means for transmitting a control signal for control. In a receiving method for receiving packetized multimedia data including a video signal transmitted from a transmitting side,
A capturing step of capturing a packetized video signal transmitted from the transmission side;
Buffering the captured video signal by writing to and reading from the memory of the buffer means;
A reproduction step of reproducing the buffered video signal;
Based on the time delay from input to output obtained from the difference between the read address and write address of the buffer means , it is detected that the difference between the video signal capture time and the playback time is within the first time as the upper limit. A first detection step to:
A second detection step of detecting that the difference between the video signal capture time and the playback time is within a second time as a lower limit based on a time delay from input to output in the buffer means;
When it is detected that the difference between the capture time of the video signal and the playback time exceeds the first time and / or falls below the second time, the input from the buffer means to the output is detected. In order to set the difference between the signal capture time based on the time delay and the reproduction time to be between the first time and the second time, the image quality or the number of transmission frames within the unit time is set to the transmission side. And a control signal transmitting step of transmitting a control signal for control.

Images