JP3612220B2 - Person monitoring method - Google Patents

Description

なるベクトルで、Ｗ，Ｈはそれぞれ抽出領域の外接矩形の縦横サイズであり、Ｓは抽出領域面積である。又Ｔｗｕ，Ｔｗｌは外接矩形の縦サイズの、Ｔｈｕ，Ｔｈｌは外接矩形の横サイズの、Ｔｓｕ，Ｔｓｌは抽出面積のそれぞれ上限値と下限値を示す閾値であり、０≦ Ｃｆ ≦１となる。
【００２６】
この信頼度算出用のベクトルＶ，Ｖｔ については、円形度Ｒを加えて以下のようにしてもよい。

ただしＴｒ は人物としてのＲの標準的値。
【００２７】
又別手法として事後確率変換による手法を用いても良い（中井事後確率を用いた移動物体検出手法、情処研報，９４−ＣＶ−９０，ｐｐ．１−８，１９９４参照）。
【００２８】
この事後確率変換による手法の処理の流れを図４のフローチャートに示す。事象θ０を背景、事象θ１を侵入物とする。ある部分画像内の画素の輝度を観測する前のθ０、θ１の事前確率がそれぞれｗ０，ｗ１であるとき（ただしｗ０＋ｗ１＝１）、事後確率ｗ１′は事象θｋのもとでの画素輝度Ｉの条件付き確率をＰ（Ｉ｜θｋ）とするときベイズの定理から

となり、このｗ１′がある閾値Ｔより大なるものを侵入物に属する画素として抽出する。ここで、あらかじめ求めた背景のみの画像の頻度分布ｈ（ｘ，ｙ，Ｉ）から、

であり、ｐ（Ｉ｜θ１）は一般にデータサンプルが無いため、
ｐ（Ｉ｜θ０）＝１／（Ｉｍａｘ−Ｉｍｉｎ） …（５）
と推定する。又上記は入力画像が更新されるたびにｗ１＝ｗ１′と更新し、ｗ１の初期値＝１．０／２．０としてよい。
【００２９】
人物候補領域を抽出するためには、入力画像内の各画素ｉについて（２）式のｗ′ｉをもとめｗ′ｉ＞Ｔなる画素で、８連結、又は４連結の意味で連結するもののうち、その面積や、外接矩形サイズが人物として妥当なものを選択すればよい。
【００３０】
この方法によると背景領域に定常的に変動があっても、統計的に変動を吸収できる利点がある。
上記で非背景（すなわち対象物）であるための事後確率を画素ごとに求めているが、閾値との比較による抽出（２値化）の前に、確率分布を４近傍や８近傍について平滑化しても良い。これは対象物近傍についての確率場の連続性を仮定し、ノイズによる過抽出やかすれを減少させるためである。又さらなるノイズ除去のために上記２値化後に数回に渡って、膨張収縮処理を繰り返しても良い。
【００３１】
この場合にも抽出領域の信頼度Ｃ１を（１）式に基づいて算出するが、前述の（２）式のＶ，Ｖｔをここでは例えば以下のようにして算出しておく。

又は円形度も含めた場合には、

ここにＰは抽出矩形内の事後確率の正規化総和であり、以下のように求められる。
【００３２】

ただしＴｐは人物に対する上述のＰ値の標準値であり、Ｒｅｃｔは抽出外接矩形領域を示す。
【００３３】
次に図２に示した人物抽出手段１における人物領域確認手段７について説明する。人物領域確認手段７は、検知頻度計測手段９に前述した検知の頻度が設定閾値を超えているかどうかを問い合わせる。もし検知頻度が閾値Ｔｈ以下であった場合には、様々な人物の体の部分の画像が複数収集され、その情報を統計的に圧縮した辞書を用いたパターンマッチングを適用して、上記のように抽出された人物候補領域内部の画素について、特徴的部分が存在するかどうかを検出する。この様子を図５に示す。
【００３４】
このような方法として、ある一定サイズに正規化した複数の特徴部分画像をある基準で複数のクラスに分割しておき、それに基づいて正準判別する方法がある。この方法による処理の流れを図６のフローチャートに示す。先ず、以下のようなステップをとる。
【００３５】
（ｉ ）複数クラスの正規化画像群に対する級内分散Ｂ、および級間分散Ｗの算出
（ｉｉ）（Ｂ＋βＳｗ ）Φ＝｛（１−β）Ｗ＋βＩ｝ΦΛなる固有値問題を解く
ただし０＜β＜１、
Φ：固有ベクトルΦｉを列ベクトルとする固有ベクトル行列、
Λ：固有値λｉを対角要素とする固有値行列
である。
【００３６】
（ｉｉｉ ）各クラスタｉ内ベクトルを上述のΦに射影し、その平均ＭｉとΦと場合によっては分散共分散Ｖｉを辞書とする。
（ｖｉ）マッチング時には画面内を小領域ｊに分割し、その小領域ごとの画像ベクトルを上記Φに射影したベクトルＶｊと各クラスｉとのユークリッド距離やマハラノビス距離を求める。
【００３７】
上記でクラス作成時に所望対象特徴画像以外に、非対象物クラスとして、それと類似する他の画像も使用すると所望領域以外の類似領域は抽出せず、所望領域のみを抽出しやすくなる。例えば、所望対象特徴画像として複数人物の正面向きの顔、非対象物クラスとして背景の顔類似パターンを用いるなどである。
【００３８】
ここでは対象物クラスを、複数人物の顔領域の画像から作成することとし、それらを顔の向きに応じて数種類の別クラスに分割する。又非対象物クラスとして、背景中の顔類似パターンをあらかじめ人手でピックアップしておき、それらを用いる。
【００３９】
そしてステップ（ｖｉ）のマッチング結果のうち距離が小なるものからＮ番目までで、対象物クラスに属するものの距離の総和が一定値Ｔｄ以下だった場合人物領域とし確認されたものとする。この場合には第二次信頼度Ｃ２を例えば以下のように算出しておく。
【００４０】

又以上の処理終了後に、上記人物候補領域抽出手段６について説明したように抽出された人物候補領域の外側で上述のマッチングを行い、その距離が小なるものからＭ番目までのものを非対象物クラスのデータとして保存する。
【００４１】
次に図２に示した人物抽出手段１における人物候補通信手段８について説明する。前述したように検知頻度計測手段９によって、人物候補の検知頻度が一定間隔Ｔｈと比較された結果が中間サーバに伝送されると、中間サーバからはその結果に応じて以下のように人物抽出手段１での処理分担指示メッセージが送られるため、人物抽出手段１での処理はそれに従う。
【００４２】
（ｉ ）検知頻度≧Ｔｈのとき
人物抽出手段１での処理は前述の人物候補領域抽出手段６の処理のみとなる。
（ｉｉ）検知頻度＜Ｔｈのとき
人物抽出手段１での処理は人物候補領域抽出手段６かつ人物領域確認手段７の処理となる。
【００４３】
ここで（ｉ ）の場合には、この人物候補通信手段８の出力結果は、例えば、上記人物候補領域抽出手段６が抽出した抽出領域（周囲の外接矩形）座標、抽出時刻、抽出領域の信頼度Ｃ１であり、（ｉｉ）の場合には、人物候補通信手段８の出力結果は、例えば、上記（ｉ ）の抽出領域（周囲の外接矩形）座標と時刻に加えて、抽出領域の信頼度Ｃを以下のように算出して伝送してもよい。
【００４４】
Ｃ＝ｍｉｎ（Ｃ１，Ｃ２） …（８）
又（ｉｉ）の場合には、ステップ（ｖｉ）の結果から以下の情報を抽出して伝送する。これは最適シーンのカメラを選択するための情報で、人物候補領域内での上記マッチング結果領域のうち、距離小なるものからＮ番目までのマッチング距離、およびそれらの各々とマッチした辞書カテゴリ名、各々の画面内での位置、および人物候補領域外側で上述のマッチングを行い、その距離が小なるものからＭ番目までのものの非対象物クラスのデータとしてマッチング距離、およびそれらの各々とマッチした辞書カテゴリ名、各々の画面内での位置も伝送する。
【００４５】
又さらに上述の伝送された対象抽出結果に応じて、中間サーバ手段２から送られる画像伝送メッセージにしたがって、後述のような所望の画像伝送動作を行う。
【００４６】
次に図１の中間サーバ手段２について説明する。図７は中間サーバ手段２の構成を示す図である。この中間サーバ手段２は人物抽出通信手段１０、人物抽出補助手段１１、最適撮像選択手段１２、サーバ通信手段１３および抽出人物画像蓄積手段１４からなる。以下には、そのそれぞれの動作を示す。
【００４７】
人物抽出通信手段１０は、各人物抽出手段１での検知頻度に応じた人物抽出機能の負荷分担、および同じく人物抽出手段１よりの認識確信度と中間サーバ手段２の通信負荷とによる抽出画像伝送の制御を行い、その結果として人物抽出手段手段１から送られてくる人物抽出画像を受け取る。受信した人物抽出画像は抽出人物画像蓄積手段１４に画面内での位置情報、信頼度、抽出時刻とともに保存される。
【００４８】
この通信処理は例えば以下のような手順で実行される。この人物抽出通信手段１０の処理の流れを図８のフローチャートに示す。
（ｉ ）人物抽出手段内部状態の取得と人物抽出方法の指示
各人物抽出手段１から内部状態として、一時保存画像数Ｎｓと検知頻度Ｈが閾値Ｔｈを超えているかどうかを取得し、前記人物候補通信手段８の説明であった検知頻度に応じて、人物抽出手段１内の処理を人物候補領域抽出手段６のみの処理とするのか、人物領域確認手段７をも行わせるのかを指示する。このようにすることによって、人物候補の確認までを必要とするが、監視領域内への人物の侵入頻度の高い場所においても検知もれを防ぐことを可能とする。又、人物候補確認までを必要としないような環境においては、常に人物候補領域抽出手段６のみの処理とさせることも可能である。
【００４９】
なお上記で人物領域確認手段７の処理をも中間サーバ手段２内部で行わせる場合には、前述の人物領域確認手段７で述べたように、保存されていた人物候補領域の外側でのマッチング結果（非対象物クラスデータ）の周囲座標も伝送する。
【００５０】
（ｉｉ）認識確信度と通信負荷とに応じた画像伝送制御
次に各人物抽出手段１から上記人物候補通信手段８で示したように抽出領域、抽出時刻、抽出領域の信頼度Ｃｋｌが送られるが、これと各人物抽出手段１での検知頻度Ｈｋ、通信トラフィックとから例えば以下のような方式にしたがって画像伝送の制御すなわち伝送画像領域Ｒｋを求める。ただし、ｋは人物抽出手段番号、ｌは一つの人物抽出手段内での抽出領域番号で０≦ｌ≦Ｌとする。
【００５１】

ただし抽出領域をＲｋｌとし、Ｔｋｌ＞画面全体のときＴｋｌ＝画面全体とする。又、Ｕは集合の和を示す。
【００５２】

ここにＳＨは通信トラフィックをあらわす量であり、例えば全人物抽出手段での検知頻度の総和を用いて以下のようにしてもよい。
【００５３】

又Ｃｋｌは各抽出領域ごとの信頼度、Ｈｋ は各人物抽出手段の検知頻度、Ｋは各人物抽出手段のアクセス数である。
【００５４】
さらに、前記（ｉ ）での非対象クラスの画像を伝送する場合には、前述の検出結果の一時保存画像と同一の扱いとする。
次に図７の人物抽出補助手段１１について説明する。図９は人物抽出補助手段１１の処理の流れを示すフローチャートである。人物抽出補助手段１１は、前述した図２の人物候補通信手段８の説明で（ｉ ）検出頻度≧Ｔｈのときに、人物領域確認手段７とまったく同一の処理を行う。
【００５５】
又人物確認における非対象クラスに対する辞書更新用データ数の前回辞書更新以降での総和が伝送画像中から一定値（Ｎｓ×α）個以上収集できたときには、本発明による人物確認用の辞書の再構成を行う。
【００５６】
ここにＮｓ従来からの非対象サンプル数、α０〜１の間の定数であり、再構成用サンプルとしては、上述の新規サンプルＮｓ×α個の他に従来サンプルの中からＮｓ×（１−α）個をランダムに選択して前記（２）（ｂ）人物領域確認手段７の（ｉ）〜（ｉｉｉ ）までの処理によって新規辞書を作成する。
【００５７】
次に図７の最適撮像選択手段１２について説明する。図１０は本発明による最適撮像選択手段１２の処理の流れを示すフローチャートである。
図２の人物候補通信手段８の説明中（ｉｉ）で述べたように、人物領域確認手段７のステップ（ｖｉ）のマッチング結果として人物候補領域内でのマッチング結果領域のうち、距離小なるものからＮ番目までのマッチング距離、およびそれらの各々とマッチした辞書カテゴリ名、各々の画面内での位置が最適撮像選択手段１２に伝送されてくる。
【００５８】
ここで最適性の定義として、上記マッチング結果のうち位置の分布がまとまっており、かつマッチング辞書カテゴリとして正面に近いものとの距離が小なるものとする。これを表現する特徴量Ｆ（最適人物撮像特徴量）としては、例えば以下のようなものを使用する。
【００５９】

ここにσは各マッチング結果領域の座標分布の標準偏差であり、Ｄｉはｉ番目にマッチング距離が小さかった分割ウィンドウと辞書との距離である。Ａ，Ｂ，Ｃｊは重み定数で、特にＣｊは上述のｉ番目ウィンドウがマッチした辞書ｊの顔の向きに応じた重みであり、正面に近いほど大きく設定する。
【００６０】
そして、この中間サーバ手段２に接続されている複数（ｋ＝０〜Ｋ個）の人物抽出手段１からの人物抽出領域のうちで、互いに物理的な位置（実世界での座標）がオーバラップするような各領域を同一物体と対応付けし、それらについての上述Ｆ値のうちで最大の値Ｆｋを有する人物抽出手段ｋからの映像を最適なものと選択する。ここで物理的な位置は、あらかじめ計測しておく撮像環境（人物抽出手段の撮像系の向き、俯角、ズーム）と人物抽出時に検出された人物候補領域の座標とから求める。
【００６１】
なお、上記Ａ，Ｂ，Ｃｊの決定については、さまざまな手法が考えられるが、例えばあらかじめ複数のサンプル画像において最適なものを教示しておき、それらサンプルにおけるσｘ，σｙ，Ｄｉを用いて最適と教示されたもののＦ値が他のものより大きいと仮定して決定する。すなわち最適と教示されたものがサンプルｋ′としたとき、

を最大化するように学習を行なって、上記Ａ，Ｂ，Ｃｊを決定する。
【００６２】
図７のサーバ通信手段１３は、最適撮像選択手段１２において最適人物画像系列が準備された場合には、この系列画像をサーバに伝送する。
最後に図１のサーバ手段３について説明する。サーバ手段３は上記サーバ通信手段１３から送信されてくる最適画像系列を受け取り、ｈｔｔｐプロトコルに準拠した形式に変換し伝送する。
【００６３】
このとき、上記系列内に冗長な画像が含まれており、かつ本サーバ手段３への通信トラフィック量が多い場合には、本発明によるサーバ手段３はそのうちのいずれかのみを残して、他は除去して伝送する。
【００６４】
ここで上述の冗長性としては、連続する最適画像系列のフレーム同士での類似度の高いものという意味で例えば、２フレーム間の画像差分量の総和が一定閾値以下の場合や、２フレーム間での画像の相関値が別の一定閾値以上の場合など一般的なものを使用できる。
【００６５】
ここで、サーバへの通信トラフィック量としては、本サーバに対するｈｔｔｐアクセスでの一定時間内パケット数Ｐｈと中間サーバ２との画像、および抽出属性の伝送に用いられる一定時間内パケット数Ｐｍとの総和として定義できる。
【００６６】
なお上記において、中間サーバ２から得られる最適画像系列には、位置や抽出時刻といった属性が含まれるが、本サーバ手段３内では、これに加えて、領域内での色空間での画素値の頻度分布、上記位置と抽出時刻から求められる平均移動ベクトルを計算して付与し検索用タグ情報とする。さらに上記最適人物画像系列中の顔領域について、図２の人物領域確認手段７の（ｖｉ）に述べた顔辞書との射影値ベクトルをも属性値として付与しておくことで、検索対象人物の画像が与えられると、その人物の顔を過去の監視画像中の最適系列画像から検索して提示することが可能となる。
【００６７】
【発明の効果】
本発明によれば、各処理ごとの機能分担による応答性向上とコストの削減が図れる。即ち、人物監視に最も基本的機能でかつリアルタイム処理を要する人物抽出機能をテレビカメラ内蔵とし、それらテレビカメラからの抽出映像の最適性を評価する中間サーバ、および検索タグ付けなど処理時間は要するがリアルタイム性が必要でない処理を行なうサーバとに分担させ、それぞれの間はデジタル化され要約された情報を伝送することで、伝送、蓄積の応答性とコストの削減が可能となる。
【００６８】
又本発明によれば、人物特徴部分検出による誤抽出可能性の低減と最適映像の検出が可能となる。即ち、人物抽出機能において、人物候補領域抽出を行なった後、その領域内に人物の特徴的部分（顔等）が写っているかどうかを評価するため、誤って人物の侵入以外のイベントを検出する頻度を低減させることが可能となる。又その際に抽出された人物特徴部分の画面内への出現の仕方を用いて、その画像フレームが後の監視員による確認に良いものなのかどうかを評価し、複数カメラの複数フレームから最適な画像系列を選択可能となる。
【００６９】
更に本発明によれば、冗長映像の除去と負荷に応じた動的処理分担によるさらなる伝送、蓄積効率の向上が図られる。即ち、上記で選択された見え方として最適な画像のフレーム間での冗長性を評価して、冗長フレームを除去することで、伝送、蓄積量を低減させ、効率を向上することが可能である。又侵入者の検出頻度や、伝送負荷に応じて各処理階層（カメラ、中間サーバ、サーバ）での処理を自動的に再配置することによって監視システム全体としての伝送、蓄積効率を向上させることが可能となる。
従って、従来よりコストを低減し、見逃しのない人物監視が本発明により可能となる。
【図面の簡単な説明】
【図１】図１は本発明の実施例に係る全体構成図。
【図２】図２は本発明の実施例に係る人物抽出手段の構成図。
【図３】図３は人物候補領域抽出手段の第１の方法を示すフローチャート。
【図４】図４は人物候補領域抽出手段の第２の方法を示すフローチャート。
【図５】図５は本発明による人物領域確認手段を説明するための図。
【図６】図６は本発明による人物領域確認手段での処理を示すフローチャート。
【図７】図７は本発明による中間サーバ手段の構成を示す図。
【図８】図８は本発明による中間サーバ手段における人物抽出通信手段の処理を示すフローチャート。
【図９】図９は本発明による中間サーバ手段における人物抽出補助手段の処理を示すフローチャート。
【図１０】図１０は本発明による中間サーバ手段における最適撮像選択手段の処理を示すフローチャート。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to person monitoring for monitoring intruders in stores, ATM installation facilities, offices, etc. Method About.
[0002]
[Prior art]
In the field of monitoring intruders in stores, ATM installation facilities and offices, methods using image transmission, monitoring by monitoring image transmission, VTR recording of monitoring images, monitoring by image processing recognition, and a combination thereof are used. I came. Hereinafter, these methods will be described.
[0003]
In monitoring by monitoring image transmission, an analog ITV camera is installed in a monitoring environment, an analog image is generally transmitted to a monitoring center at a remote location, and a monitoring person installed in the center constantly monitors the monitoring image.
[0004]
In this case, generally, images from a plurality of cameras are transmitted, and the monitor monitors the image display consciously or by switching at regular intervals. In some cases, an intruder detection signal from an intrusion sensor such as an infrared ray or an ultrasonic wave installed in a monitoring environment is used as an opportunity for switching the screen. Some images are not converted into analog images but are transmitted after being converted into compressed data by Jpeg or Mpeg.
[0005]
In the VTR recording of the monitoring image, an analog ITV camera is installed in the monitoring environment, and an image is recorded at regular intervals on a VTR apparatus installed in a remote place or a place that cannot be seen from the monitoring environment.
The monitoring based on the image processing recognition is a notification or VTR activation by detecting a change of an image in a monitoring environment or a person candidate area in the image.
[0006]
[Problems to be solved by the invention]
The conventional techniques as described above have the following problems.
First of all, the problem with monitoring by monitoring image transmission is that the processing mechanism is simple, but in order to perform analog image transmission, there are additional facilities such as laying a dedicated transmission path and installing a booster to guarantee signal degradation due to transmission. Necessary. Therefore, the cost for these facilities is enormous, and it is necessary to always have an observer at the center.
[0007]
On the other hand, in systems that transmit compressed image data, it is possible to use a general digital line instead of an analog dedicated line for reasons such as less data than the above and no signal degradation due to transmission. Since a lot of other information is also transmitted, the problem of constant deployment of observers cannot be solved.
[0008]
To describe the problem with VTR recording of surveillance images, this system basically records intruder images separately from emergency calls, and is used later for situation estimation and intruder identification. However, in some cases, recording at regular intervals cannot actually record an important video as an intrusion event.
[0009]
The problem with monitoring by image processing is that if the intruder detection rate is reasonable, this system requires transmission and storage processing only at the time of intrusion. It is possible to make it low. In addition, the necessity of constant deployment can be reduced by checking the transmitted and accumulated images only when an event occurs.
[0010]
However, after detecting an event, the nature of the transmitted or accumulated image is often not good for the observer to confirm. For example, there is no characteristic part such as a face in the person area in the monitoring video, the image is too small, or the image quality is deteriorated due to the compression.
[0011]
Therefore, the present invention Providing a person monitoring method that can reduce costs and monitor people without missing them Objective.
[0012]
[Means for Solving the Problems]
In the method of performing person monitoring using a plurality of image pickup means provided for picking up images of a monitoring area from different positions, and a first server and a second server receiving the images from the respective image pickup means, Extract human candidate images from captured images at each imaging means Share the processing that requires real-time processing, and extract the extracted person candidate images The first server evaluates the appearance of the person in the plurality of person candidate images transmitted from the respective imaging means, and the person candidate image having the optimum person appearance is transmitted to the first server. Choice Share the costly processing of the selected person candidate images Output to the second server as a monitoring image, and the second server receives the monitoring image output by the first server; Shares processing that requires processing time to generate image search keys but does not require real-time processing. Find the image difference amount between two consecutive frames, and if the sum is less than a certain value, remove only one of them and remove the other image, convert only the remaining image to http protocol and transmit To do.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, an apparatus for monitoring an intruder in a certain monitoring area using the present system will be described.
[0017]
First, the overall configuration and processing outline of this embodiment will be described. The overall configuration of the person monitoring system according to the present invention is shown in FIG. This system comprises person extraction means 1, intermediate server means 2, and server means 3. The person extraction means 1 extracts a person candidate area from the input moving image, and the intermediate server means 2 shows an image that is optimally viewed. The server unit 3 generates a key for image search and provides a user interface using http (hyper text transfer protocol). Each means 1, 2, 3 is connected by wired or wireless digital communication.
[0018]
Accordingly, the person extraction unit 1 or the intermediate server unit 2 checks and looks at the person feature for the person candidate area in the input image detected by the one or more person extraction units 1. But The optimum frame is selected and the camera (this corresponds to the person extracting means 1) is selected. The server means 3 temporarily stores the optimum human image series obtained from the intermediate server means 2, and then The image series is transmitted.
[0019]
At this time, For example For the detected person series, an image search key such as a moving direction, a color, a way of movement, and a similarity with the reference person image is created, and only a desired image is transmitted according to a request from the user.
[0020]
As described above, according to the state of the transmission path between the person extracting unit 1 and the intermediate server 2 and between the intermediate server unit 2 and the server unit 3, the number of intrusion events of the target person, and the like, the sharing range of each process as appropriate change. For example, the person extraction means 1 performs extraction and confirmation of a person candidate area, but when the number of events per unit time is large, the person candidate confirmation process is performed only in a relatively rough manner, such as a face described later. Those having a high processing cost such as confirmation by the characteristic portion are shared by the intermediate server means 2.
[0021]
Hereinafter, the configuration and processing of the person extracting means 1, the intermediate server means 2, and the server means 3 will be described.
FIG. 2 is a diagram showing the configuration of the person extracting means 1. Digital image data input every several frames by the image input means 4 composed of a color or monochrome television camera and an A / D converter is once stored in the person detection image storage means 5, and this will be described later. The person candidate area extraction unit 6 detects the approximate area of the intruding person, and the person area confirmation unit 7 determines whether or not a characteristic area pattern (for example, face, hand, foot, etc.) exists as a person within the approximate area. The person candidate area image and the confirmation result are transmitted to the intermediate server by Ethernet or IEEE 1394 using a protocol such as Tcp / IP. If the input image is color as described above, only the luminance component is extracted and stored in the person detection image storage means 5.
[0022]
In addition, the person extraction means 1 measures the number of extractions of the above-described target person area within a certain time, that is, the detection frequency, by the detection frequency measurement means 9 in the person extraction means 1, and this exceeds a preset threshold value. If there is, the processing of this means should not take time, and the processing of the person area confirmation means 7 which generally requires processing costs is not performed, and the sharing of processing is requested to an intermediate server described later.
[0023]
Hereinafter, the person candidate area extracting means 6, the person area confirming means 7, and the person candidate communicating means 8 will be described as functions characteristic of the person extracting means 1.
As the process of the person candidate region extraction unit 6, any of several methods as described below can be used.
[0024]
For example, a difference for each pixel is performed on consecutive sampled images (frames), a binarized region is extracted with a certain threshold, and the area and spatially connected in the sense of 4-connected or 8-connected This is a method of detecting a circumscribed rectangle size that is appropriate as a human candidate. The flow of this process is shown in the flowchart of FIG. In this case, the binarization threshold can be determined by analyzing the frequency distribution for each luminance of the difference image and determining a value that maximizes the variance ratio (automatic based on Otsu, discrimination and least squares criterion). (See threshold selection method, theory of theory, Vol. J63-D, No. 4, pp. 349-356, 1980). Here, before the analysis of the connected area, the extraction area may be subjected to several expansion and contraction processes to reduce the influence of over-extraction and “blur” due to noise. In this case, the primary reliability C1 of the extraction area is calculated as follows, for example.
[0025]

W and H are the vertical and horizontal sizes of the circumscribed rectangle of the extraction region, and S is the extraction region area. Also, Twu and Twl are the vertical size of the circumscribed rectangle, Thu and Thl are the horizontal size of the circumscribed rectangle, and Tsu and Tsl are thresholds indicating the upper limit value and the lower limit value of the extraction area, respectively, and 0 ≦ Cf ≦ 1.
[0026]
The reliability calculation vectors V and Vt may be as follows with the circularity R added.

Where Tr is the standard value of R as a person.
[0027]
As another method, a method based on posterior probability conversion may be used (see moving object detection method using Nakai posterior probability, Information Processing Research Bulletin, 94-CV-90, pp. 1-8, 1994).
[0028]
The flow of the process of the technique by this posterior probability conversion is shown in the flowchart of FIG. Event θ0 is assumed to be a background, and event θ1 is assumed to be an intruder. When the prior probabilities of θ0 and θ1 before observing the luminance of the pixels in a partial image are w0 and w1, respectively (where w0 + w1 = 1), the posterior probability w1 ′ is the pixel luminance I under the event θk. From Bayes' theorem when conditional probability is P (I | θk)

Thus, a pixel whose w1 ′ is greater than a certain threshold value T is extracted as a pixel belonging to the intruder. Here, from the frequency distribution h (x, y, I) of the image of the background only obtained in advance,

Since p (I | θ1) generally has no data sample,
p (I | θ0) = 1 / (Imax−Imin) (5)
Estimated. In the above, w1 = w1 ′ is updated every time the input image is updated, and the initial value of w1 may be 1.0 / 2.0.
[0029]
In order to extract a human candidate region, for each pixel i in the input image, a pixel with w′i> T obtained from w′i in equation (2) and connected in the sense of 8-connected or 4-connected. It is sufficient to select a person whose area or circumscribed rectangle size is appropriate as a person.
[0030]
According to this method, there is an advantage that even if there is a steady variation in the background region, the variation can be absorbed statistically.
In the above, the posterior probability for non-background (that is, the object) is obtained for each pixel, but before the extraction (binarization) by comparison with the threshold value, the probability distribution is smoothed for 4 neighborhoods and 8 neighborhoods. May be. This is to reduce over-extraction and blur due to noise, assuming continuity of the random field around the object. In order to further eliminate noise, the expansion / contraction process may be repeated several times after the above binarization.
[0031]
In this case as well, the reliability C1 of the extraction region is calculated based on the equation (1). Here, V and Vt in the above equation (2) are calculated as follows, for example.

Or when including the circularity,

Here, P is a normalized sum of the posterior probabilities in the extraction rectangle, and is obtained as follows.
[0032]

However, Tp is a standard value of the above-mentioned P value for a person, and Rect indicates an extracted circumscribed rectangular area.
[0033]
Next, the person area confirmation means 7 in the person extraction means 1 shown in FIG. 2 will be described. The person area confirmation unit 7 inquires of the detection frequency measurement unit 9 whether the above-described detection frequency exceeds the set threshold value. If the detection frequency is less than or equal to the threshold Th, a plurality of images of various human body parts are collected, and pattern matching using a dictionary in which the information is statistically compressed is applied as described above. It is detected whether or not there is a characteristic part for the pixels inside the human candidate area extracted in (1). This is shown in FIG.
[0034]
As such a method, there is a method in which a plurality of feature partial images normalized to a certain size are divided into a plurality of classes based on a certain standard, and canonical discrimination is performed based on the divided classes. The flow of processing by this method is shown in the flowchart of FIG. First, the following steps are taken.
[0035]
(I) Calculation of intra-class variance B and inter-class variance W for multiple classes of normalized image groups
(Ii) Solve the eigenvalue problem of (B + βSw) Φ = {(1-β) W + βI} ΦΛ
Where 0 <β <1,
Φ: an eigenvector matrix having eigenvectors Φi as column vectors,
Λ: Eigenvalue matrix with eigenvalue λi as a diagonal element
It is.
[0036]
(Iii) Each intra-cluster i vector is projected onto the above-mentioned Φ, and the average Mi and Φ and, in some cases, the variance covariance Vi are used as a dictionary.
(Vi) At the time of matching, the screen is divided into small areas j, and the Euclidean distance and the Mahalanobis distance between the vector Vj obtained by projecting the image vector of each small area onto the Φ and each class i are obtained.
[0037]
When other images similar to the non-object class are used in addition to the desired target feature image at the time of creating the class, a similar region other than the desired region is not extracted and only the desired region can be easily extracted. For example, a face-facing pattern of a plurality of persons is used as the desired target feature image, and a face-like pattern in the background is used as the non-target class.
[0038]
Here, the object class is created from the images of the face areas of a plurality of persons, and these are divided into several different classes according to the orientation of the face. As the non-object class, face-like patterns in the background are previously picked up manually and used.
[0039]
Then, it is assumed that the person area is confirmed when the sum of the distances of the matching results in step (vi) from the smallest distance to the Nth and belonging to the object class is equal to or less than a predetermined value Td. In this case, the secondary reliability C2 is calculated as follows, for example.
[0040]

Further, after the above processing is completed, the above-described matching is performed outside the extracted person candidate area as described for the person candidate area extracting means 6, and the objects whose distance is small to Mth are non-objects. Save as class data.
[0041]
Next, the person candidate communication means 8 in the person extraction means 1 shown in FIG. 2 will be described. As described above, when the result of comparison of the detection frequency of the human candidate with the constant interval Th is transmitted to the intermediate server by the detection frequency measuring unit 9, the human server extracts the person according to the result as follows. Since the process sharing instruction message in 1 is sent, the process in the person extracting means 1 follows it.
[0042]
(I) When detection frequency ≧ Th
The processing by the person extracting means 1 is only the processing of the person candidate area extracting means 6 described above.
(Ii) When detection frequency <Th
The processing in the person extraction means 1 is the processing of the person candidate area extraction means 6 and the person area confirmation means 7.
[0043]
In the case of (i), the output result of the person candidate communication means 8 is, for example, the extracted area (surrounding circumscribed rectangle) coordinates extracted by the person candidate area extracting means 6, the extraction time, and the reliability of the extraction area. In the case of (ii), the output result of the person candidate communication means 8 is, for example, the reliability of the extraction area in addition to the extraction area (surrounding rectangle) coordinates and time of (i) above. C may be calculated and transmitted as follows.
[0044]
C = min (C1, C2) (8)
In the case of (ii), the following information is extracted from the result of step (vi) and transmitted. This is information for selecting the camera of the optimum scene. Among the matching result areas in the person candidate area, the matching distance from the smallest distance to the Nth, and the dictionary category name that matches each of them, The above-mentioned matching is performed outside the candidate area and the position in each screen, and the matching distance as non-object class data from the smallest to the Mth, and a dictionary that matches each of them The category name and the position in each screen are also transmitted.
[0045]
Further, a desired image transmission operation as described later is performed in accordance with the image transmission message sent from the intermediate server means 2 in accordance with the transmitted object extraction result.
[0046]
Next, the intermediate server means 2 in FIG. 1 will be described. FIG. 7 is a diagram showing the configuration of the intermediate server means 2. The intermediate server unit 2 includes a person extraction communication unit 10, a person extraction auxiliary unit 11, an optimum imaging selection unit 12, a server communication unit 13, and an extracted person image storage unit 14. Below, each operation | movement is shown.
[0047]
The person extraction communication means 10 shares the load of the person extraction function according to the detection frequency of each person extraction means 1 and also extracts image transmission based on the recognition certainty from the person extraction means 1 and the communication load of the intermediate server means 2. As a result, the person extraction image sent from the person extraction means 1 is received. The received person extracted image is stored in the extracted person image accumulating unit 14 together with position information on the screen, reliability, and extraction time.
[0048]
This communication process is executed by the following procedure, for example. The processing flow of the person extraction communication means 10 is shown in the flowchart of FIG.
(I) Acquisition of person extraction means internal state and instruction of person extraction method
As an internal state from each person extraction means 1, whether or not the number of temporarily stored images Ns and the detection frequency H exceeds a threshold Th is acquired, and the person extraction is performed according to the detection frequency described in the person candidate communication means 8. It is instructed whether the processing in the means 1 is to be performed only by the person candidate area extraction means 6 or the person area confirmation means 7 is also performed. By doing so, it is necessary to confirm the person candidate, but it is possible to prevent a detection leak even in a place where the frequency of intrusion of the person into the monitoring area is high. Further, in an environment where it is not necessary to confirm the person candidate, it is possible to always perform only the process of the person candidate area extracting means 6.
[0049]
When the processing of the person area confirmation unit 7 is also performed inside the intermediate server unit 2 as described above, as described in the person area confirmation unit 7 above, the matching result outside the stored person candidate area The peripheral coordinates of (non-object class data) are also transmitted.
[0050]
(Ii) Image transmission control according to recognition confidence and communication load
Next, as shown by the person candidate communication means 8 from each person extraction means 1, the extraction area, the extraction time, and the reliability Ckl of the extraction area are sent, and this and the detection frequency Hk in each person extraction means 1, communication From the traffic, for example, image transmission control, that is, a transmission image area Rk is obtained according to the following method. Here, k is a person extraction means number, and l is an extraction area number in one person extraction means, where 0 ≦ l ≦ L.
[0051]

However, the extraction region is Rkl, and when Tkl> the entire screen, Tkl = the entire screen. U represents the sum of the set.
[0052]

Here, SH is an amount representing communication traffic. For example, SH may be used as follows by using the sum of the detection frequencies of all person extraction means.
[0053]

Ckl is the reliability of each extraction area, Hk is the detection frequency of each person extraction means, and K is the number of accesses of each person extraction means.
[0054]
Further, when the image of the non-target class in (i) is transmitted, it is handled in the same manner as the temporarily stored image of the detection result described above.
Next, the person extraction assisting means 11 in FIG. 7 will be described. FIG. 9 is a flowchart showing the flow of processing of the person extraction assisting means 11. The person extraction assistance means 11 performs exactly the same processing as the person area confirmation means 7 when (i) detection frequency ≧ Th in the description of the person candidate communication means 8 of FIG.
[0055]
When the sum of the number of dictionary update data for the non-target class in person confirmation since the last dictionary update can be collected from the transmission image more than a certain value (Ns × α), the person confirmation dictionary according to the present invention can be regenerated. Make the configuration.
[0056]
Here, Ns is a conventional non-target number of samples, a constant between α0 and 1. As reconstruction samples, in addition to the above-mentioned new samples Ns × α, Ns × (1−α ) Are randomly selected, and a new dictionary is created by the processes (i) to (iii) of the (2) (b) person area confirmation means 7.
[0057]
Next, the optimum imaging selection unit 12 in FIG. 7 will be described. FIG. 10 is a flowchart showing the processing flow of the optimum imaging selection means 12 according to the present invention.
As described in (ii) in the explanation of the person candidate communication means 8 in FIG. 2, the matching result area in the person candidate area having a smaller distance as the matching result in the step (vi) of the person area confirmation means 7 To the Nth matching distance, the dictionary category name matched with each of them, and the position in each screen are transmitted to the optimum imaging selection means 12.
[0058]
Here, as the definition of the optimality, it is assumed that the position distribution of the matching results is collected and the distance from the matching dictionary category near the front is small. As the feature amount F (optimal person imaging feature amount) that expresses this, for example, the following is used.
[0059]

Here, σ is the standard deviation of the coordinate distribution of each matching result area, and Di is the distance between the divided window where the matching distance is the smallest and the dictionary. A, B, and Cj are weight constants. In particular, Cj is a weight according to the orientation of the face of the dictionary j matched with the above-mentioned i-th window, and is set to be larger as it is closer to the front.
[0060]
Of the person extraction areas from the plurality (k = 0 to K) of person extraction means 1 connected to the intermediate server means 2, physical positions (coordinates in the real world) overlap each other. Such regions are associated with the same object, and an image from the person extracting means k having the maximum value Fk among the above-described F values is selected as the optimum one. Here, the physical position is obtained from the imaging environment (the orientation of the imaging system of the person extraction means, the depression angle, zoom) and the coordinates of the person candidate area detected at the time of person extraction.
[0061]
Various methods can be considered for the determination of A, B, and Cj. For example, an optimum one is taught in advance in a plurality of sample images, and the optimum is determined using σx, σy, and Di in the samples. It is determined assuming that the F value of what is taught is greater than the others. That is, when the optimal teaching is sample k ′,

Learning is performed so as to maximize A, B, and Cj.
[0062]
When the optimum human image series is prepared by the optimum imaging selection means 12, the server communication means 13 in FIG. 7 transmits this series image to the server.
Finally, the server means 3 in FIG. 1 will be described. The server means 3 receives the optimum image sequence transmitted from the server communication means 13, converts it into a format compliant with the http protocol, and transmits it.
[0063]
At this time, when a redundant image is included in the series and the communication traffic amount to the server means 3 is large, the server means 3 according to the present invention leaves only one of them, Remove and transmit.
[0064]
Here, the above-mentioned redundancy means that the similarity between frames of the continuous optimum image sequence is high, for example, when the sum of image difference amounts between two frames is less than a certain threshold or between two frames. A general image can be used, for example, when the correlation value of the image is equal to or greater than another certain threshold value.
[0065]
Here, the amount of communication traffic to the server is the sum of the number of packets Ph within a certain period of time for http access to this server and the number of packets Pm within a certain period of time used for transmission of images with the intermediate server 2 and extraction attributes. Can be defined as
[0066]
In the above, the optimum image sequence obtained from the intermediate server 2 includes attributes such as position and extraction time. In the server means 3, in addition to this, the pixel value of the color space in the region is added. An average movement vector obtained from the frequency distribution, the position, and the extraction time is calculated and added to obtain search tag information. Further, by assigning a projection value vector with the face dictionary described in (vi) of the person area confirmation means 7 in FIG. 2 as an attribute value for the face area in the optimum human image series, Given an image, the person's face can be retrieved from the optimum sequence image in the past monitoring image and presented.
[0067]
【The invention's effect】
According to the present invention, it is possible to improve responsiveness and reduce costs by sharing functions for each process. In other words, the person extraction function that is the most basic function for person monitoring and requires real-time processing is incorporated in the TV camera, and an intermediate server that evaluates the optimality of the extracted video from the TV camera and processing time such as search tagging are required. By sharing the information that has been digitized and summarized between the servers that perform processing that does not require real-time processing, it is possible to reduce transmission and storage responsiveness and cost.
[0068]
In addition, according to the present invention, it is possible to reduce the possibility of erroneous extraction and to detect an optimal video by detecting a person feature portion. In other words, in the person extraction function, after extracting a human candidate area, an event other than the intrusion of a person is erroneously detected in order to evaluate whether or not a characteristic part (face, etc.) of the person is reflected in the area. The frequency can be reduced. Also, using the method of appearance of the person feature extracted at that time in the screen, it is evaluated whether the image frame is good for confirmation by a later monitor, and the optimal from the multiple frames of multiple cameras. An image series can be selected.
[0069]
Furthermore, according to the present invention, it is possible to further improve transmission and storage efficiency by removing redundant video and sharing dynamic processing according to the load. That is, it is possible to reduce the amount of transmission and storage and improve the efficiency by evaluating the redundancy between the frames of the image most suitable as the appearance selected above and removing the redundant frames. . Also, the transmission and storage efficiency of the entire monitoring system can be improved by automatically rearranging the processing at each processing layer (camera, intermediate server, server) according to the detection frequency of intruders and the transmission load. It becomes possible.
Accordingly, it is possible to monitor a person with a cost reduced and not missed by the present invention.
[Brief description of the drawings]
FIG. 1 is an overall configuration diagram according to an embodiment of the present invention.
FIG. 2 is a block diagram of a person extracting means according to an embodiment of the present invention.
FIG. 3 is a flowchart showing a first method of person candidate region extraction means;
FIG. 4 is a flowchart showing a second method of person candidate region extraction means;
FIG. 5 is a diagram for explaining a person area confirmation unit according to the present invention.
FIG. 6 is a flowchart showing processing in a person area confirmation unit according to the present invention.
FIG. 7 is a diagram showing a configuration of intermediate server means according to the present invention.
FIG. 8 is a flowchart showing processing of a person extraction communication unit in the intermediate server unit according to the present invention.
FIG. 9 is a flowchart showing processing of a person extraction assisting unit in the intermediate server unit according to the present invention.
FIG. 10 is a flowchart showing processing of optimum imaging selection means in the intermediate server means according to the present invention.

Claims (1)

In a method of performing person monitoring using a plurality of imaging means provided for imaging a video of a monitoring area from different positions, and a first server and a second server that receive video from each imaging means,
In each of the imaging means, a process that requires real-time processing for extracting a person candidate image from a captured image is shared, and the extracted person candidate image is transmitted to the first server,
The first server evaluates the appearance of the person in the plurality of person candidate images transmitted from the respective imaging means, and shares the costly process of selecting the person candidate image with the optimum person appearance. Outputting the selected person candidate image to the second server as a monitoring image;
The second server receives the monitoring image output by the first server , shares processing that requires processing time for generating an image search key but does not require real-time processing, and further between two consecutive frames The image difference amount is calculated, and when the sum of the images is less than a certain value, only one of them is removed and the other images are removed, and only the remaining image is converted to the http protocol and transmitted. Person monitoring method characterized by this.

Images