JP2004112287A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable efficient identification of an object, for example, in a moving image in an image processing apparatus for processing an image. <P>SOLUTION: An image information acquisition means 12 increases resolution in steps and at the same time reduces an image range and acquires image information based on the image to be processed. Then, an image information processing means 12 performs processing based on the image information acquired by the image information acquisition means 12. Additionally, the image information acquisition means 12 specifies the pixel information used as a reference in the next step based on the pixel information acquired in a specific step, and sets an image range in the next step with the specified pixel information as the reference. <P>COPYRIGHT: (C)2004,JPO

Description

【００６７】
ここで、（Ｘｉ、Ｙｊ）はＸ−Ｙ直交座標系の座標値を表し、リサンプリングする画素点の座標値を表す。また、Ｌは例えば横軸方向であるＸ軸方向のリサンプリング数を表し、Ｍは例えば縦軸方向であるＹ軸方向のリサンプリング数を表す。また、（Ｘ０、Ｙ０）はリサンプリングを開始する座標値を表し、ｄはリサンプリングの間隔を表す。
【００６８】
また、各段階においてリサンプリングを行う画像領域の幅をＸと表し、各段階においてリサンプリングを行う画像領域の高さをＹと表すと、Ｘ＞ｄ×Ｌ及びＹ＞ｄ×Ｍとなることが必要である。
本例では、或る段階におけるリサンプリング結果から解像度を高めてリサンプリングを行う次の段階におけるリサンプリング処理で用いられるリサンプリング開始座標値（Ｘ０、Ｙ０）を、注目して識別したい物体が画像範囲の中心となるような値に設定することにより、効率的な処理が実現される。
【００６９】
また、サンプリング間隔ｄを２のべき乗数とすると、計算が容易となる。例えば、本例のように３段階のリサンプリング処理で最大の解像度の処理となるようにするためには、第１次のリサンプリング処理の段階におけるリサンプリング間隔ｄ１＝２^２＝４とし、第２次のリサンプリング処理の段階におけるリサンプリング間隔ｄ２＝２^１＝２とし、第３次のリサンプリング処理の段階におけるリサンプリング間隔ｄ３＝２^０＝１とする態様を用いることができる。
【００７０】
更に、本例のように、横軸方向のリサンプリング数（例えば、Ｌ＝Ｘ／ｄ）及び縦軸方向のリサンプリング数（例えば、Ｍ＝Ｙ／ｄ）を各段階で共通の値とすると、画像処理を行うためのデータ領域（横軸方向の全画素数及び縦軸方向の全画素数）を各段階で一定とすることが可能であり、処理機構をシンプルにすることができる。
【００７１】
本例の場合には、それぞれの段階において元の入力画像５２に対して１／１６（＝（１／４）^２）のデータ量の処理とすることができ、３段階にわたる全体的な処理の合計においても当該値の３倍である３／１６のデータ量の処理で済ませることができる。
【００７２】
また、上述のように、リサンプリング後の画像については、例えば通常の画像処理を行うことにより、注目する対象の識別或いは判別を行う。このような識別或いは判別を行う方法としては、例えば、画素値を元にした分布ヒストグラムを使用する方法や、エッジ検出を行った後にエッジの接続から形状を認識する方法や、或いは、ニューラルネットワークを用いて判別を行う方法などを用いることができる。
【００７３】
次に、第２実施例を示す。
なお、本例の画像処理システムは、上記第１実施例の図３に示したものと同様な構成を有しており、上記第１実施例で示したのと同様な動作を行う。
上記図４に示した第１次のリサンプリング処理などでは、注目して処理すべき物体の画像データがリサンプリングの間隔の間に入ってしまって（つまり、間引きされてしまって）、当該物体を見落としてしまう可能性がある。
【００７４】
そこで、本例の画像処理システムでは、注目する物体の画素点として定めるべき画素点の座標値が決められないときには探索モードへ移行する。そして、当該探索モード時においては、同一の段階において、リサンプリング処理毎に微小間隔で注視点をずらして、例えばこれによりリサンプリング開始座標値（Ｘ０、Ｙ０）をずらして、複数回のリサンプリング処理を行う。これにより、同一の段階において、或るリサンプリング処理による情報の欠落部分を異なる時刻で処理するようにして、当該欠落部分を補間することを実現する。
【００７５】
図５には、画像（一部分）を構成する１６個の画素点について、同一の段階においてリサンプリング処理を行う起点（リサンプリング開始座標値）とする画素点の順番の一例を示してある。同図の例では、番号１、２、３、・・・、１４、１５、１６の順で、各番号の画素点を起点として順次リサンプリング処理を行う。
【００７６】
本例では、特に、第１次のリサンプリング処理では、解像度が低いため、注目して重点的に処理すべき中心点が見つからない場合がある。このような場合には、当該低解像度のまま、リサンプリング処理を行う起点を同図に示されるような順番でずらしていくことにより、リサンプリング処理する画素の重複を起こすことなく全ての画素を処理対象とすることができる。
そして、注目して処理すべき中心点が見つかった場合には、その点を中心として次の段階におけるリサンプリング処理へ移行し、これにより、リサンプリング間隔を狭めた解像度の高い画像に基づいてより確実な識別を行う。
【００７７】
このように、本例の画像処理システムでは、動物の眼球運動に似た運動に相当する処理を用いて注視点の移動を行うことにより、リサンプリングの間隔で間引きされてしまっている画素の情報を時間差で処理することが可能であり、これにより、実質的な解像度を高めることが可能である。
なお、複数の段階にわたる全体的な処理では、実質的には、重要な情報を短い時間間隔で処理する（つまり、長い時間処理する）一方で重要でない情報を長い時間間隔で処理する（つまり、短い時間処理する）ようなことが実現され、トータルの処理量をバランス良く減らすことが実現される。
【００７８】
また、例えば、動画像のように時系列的につながった複数のフレームの画像を連続して処理する場合には、画像処理において注目する中心点を各フレームの画像について求めて、異なるフレーム間における当該中心点の座標値の差分を求めることにより、注目している物体の移動を検出することや、注目している物体の移動速度などを計算することが容易にできる。
【００７９】
次に、第３実施例を示す。
なお、本例の画像処理システムは、上記第１実施例の図３に示したものと同様な構成を有しており、上記第１実施例や上記第２実施例で示したのと同様な動作を行う。
【００８０】
上記図４に示したリサンプリング処理において、１枚（１フレーム）の画像中に注目して識別したい物体が２つ以上現れる場合がある。このような場合に、本例の画像処理システムでは、例えば、上記図４に示したようにして１つの注目する物体について最終的な段階での判別を行った後に、注目するもう一方の物体に対応する座標値へリサンプリング処理の場所を変更して２次のリサンプリング処理を行い、そして、当該２次のリサンプリング処理から解像度を高めて行く処理を行う。このように、注目するもう一方の物体については、第１次のリサンプリング処理などの重複した処理を省いて、効率的に処理を行うことができる。
【００８１】
図６には、このように複数の注目する物体に関するリサンプリング処理を階層的に行う処理（階層型のリサンプリング処理）の様子の一例として、例えば上記図４に示した入力画像５２と同一の画像である入力画像６１と、例えば上記図４に示した円状の物体（ここで、第１の注目対象と言う）についてのリサンプリング画像５３、５４、５５と同一の画像であるリサンプリング画像６２、６３、６４を示してあるとともに、入力画像６１に含まれる三角形及び四角形の組み合わせからなる物体（ここで、第２の注目対象と言う）についての第２次のリサンプリング画像６５を示してある。
【００８２】
上記図６の例では、第１順目の処理として、第１の注目対象についての段階的なリサンプリング処理を第１次のリサンプリング処理から行い、その後、第２順目の処理として、第２の注目対象についての段階的なリサンプリング処理を第２次のリサンプリング処理から行う。なお、第２の注目対象は、例えば、第１順目の処理において第１の注目対象と共に注目対象として特定される。
【００８３】
また、上記のように画像の中で複数の注目する物体を設定する場合に、これら複数の注目する物体のそれぞれについて優先順位を決め、その優先順位の順番でそれぞれの注目する物体を識別処理などする構成とすることもできる。このような構成では、例えば、動画像処理のように処理時間に制約がある場合に時間切れで処理を打ち切ったときにおいても、優先順位の高い物体については正確な位置などが検出され、優先順位の低い物体については大体の位置などが検出されることとなり、実時間での動画像処理に適している。
【００８４】
次に、第４実施例を示す。
なお、本例の画像処理システムは、上記第１実施例の図３に示したものと同様な構成を有しており、上記第１実施例〜上記第３実施例で示したのと同様な動作を行う。
【００８５】
本例の画像処理システムでは、適度な間隔で２つのカメラを平行に設置して備えており、どちらのカメラの入力画像についても例えば上記第１実施例の図４で示したのと同様なリサンプリング処理を行う。
例えば、２つのカメラから得られる左右の入力画像で同じ物体を注視させた場合、対象が無限遠にある物体であれば左右の入力画像で同じ座標値が注視点となるが、対象が近い物体であるときには左右の入力画像における注視点の間に座標値の差が生じる。そして、このような座標値の差を利用して、対象となる物体との距離（例えば、概略的な距離）を計算することができる。
【００８６】
具体的な一例として、左右にある２つのカメラの中心線の間隔をＤと表し、カメラ（例えば、いずれかのカメラ）から対象となる物体までの距離をＺと表し、対象となる物体と左右のカメラとを結ぶ２つの線の間の角度差をθと表し、左右のカメラにおける対象となる物体の座標値の差が当該角度差θに比例していると近似すると、式２が成り立つことから、カメラから対象となる物体までの距離Ｚは式３のように近似される。
【００８７】
【数２】

【００８８】
【数３】

【００８９】
但し、このような近似では、対象となる物体が入力画像の中心から離れるほど、誤差が大きくなる。
また、このように対象となる物体までの距離を検出するに際して、例えば、カメラを載せる回転台座を用意すると、処理可能な角度を広げることが可能になるだけでなく、目標物が撮像素子の中心線上に近くなるようにカメラの角度を調整することにより誤差を少なくすることが可能である。
【００９０】
また、例えば、動画像を処理するに際して、移動する量が大きい物体に両方のカメラによる入力画像の注視点を合わせている場合において、ｔ（ここで、ｔは１以上の整数）フレームにおける対象物体までの距離Ｚ（ｔ）と、（ｔ＋１）フレームなどのように異なるフレームにおける対象物体までの距離（Ｚ（ｔ＋１）など）を求めることにより、対象物体の遠近方向軸での移動速度を求めることができる。
【００９１】
以上のように、本実施例に係る画像処理システムでは、映像入力手段からの映像を段階的に解像度を高めて且つ段階的に処理範囲を限定してリサンプリングして、各種の処理を行う。また、本実施例に係る画像処理システムでは、各段階においてリサンプリングされる画素のデータが固定長の２次元配列に格納され得る構成としてあり、また、リサンプリングを行う画素の間隔として２のべき乗の値を用いている。
【００９２】
従って、本実施例に係る画像処理システムでは、最初は低解像度のリサンプリング画像によっておおまかな場所を特定し、続いて、重要な物体がありそうなところに注目するようにして徐々に解像度を高めてリサンプリング処理を行い、そして、重要な物体の位置などを検出する処理を行うことにより、結果的に、注目すべき処理の中心点は密に周辺は疎にリサンプリング処理を行うことができ、このようなリサンプリング処理の結果に基づいて注目する物体に関して精度のよい情報を取得することができる。
【００９３】
また、本実施例に係る画像処理システムでは、例えば、量産されてコストダウンしている光学系デバイスや撮像デバイスを用いて、生物の網膜や眼球運動に相当する処理を実現することが可能であるため、動画像などの処理を安価に高度に行うことができる。また、本実施例に係る画像処理システムでは、注目する対象の画素点（例えば、注視点）の付近では元の画像の解像度を維持することも可能であるため、処理の中心となる注目する対象の認識率等を落とすことなく、全体的な処理量を削減することが可能である。
【００９４】
また、本実施例に係る画像処理システムでは、初期の段階のリサンプリング処理において注目する中心点が求まらなかった場合には、同一の段階において、リサンプリング処理を行う開始点となる画素点をずらすことにより、１種類のリサンプリング処理では抜けてしまう画素値の情報を複数種類のリサンプリング処理により総じて補間する。
【００９５】
従って、本実施例に係る画像処理システムでは、注目する処理の中心点が見つからなかった場合などには、フレーム毎に微少にリサンプリング点をずらして、或るリサンプリングの隙間の画素値を他のリサンプリングで処理することにより、解像度を高めずに注目する処理の中心点を特定することが可能である。このように、本実施例に係る画像処理システムでは、生物の眼球運動に相当する処理が可能であり、動画像などの処理を安価に高度に行うことができる。
【００９６】
また、本実施例に係る画像処理システムでは、リサンプリング処理や識別処理を注目する複数の対象について階層的に行うことにより、１枚の画像において複数の対象物体の識別を行う。また、本実施例に係る画像処理システムでは、このような複数の対象についての処理を、それぞれの対象に付与された優先順位の順番で行う。
【００９７】
従って、本実施例に係る画像処理システムでは、１枚の画像中に複数の識別したい物体があるような場合においても、階層型のリサンプリング処理を行うことにより、これら複数の物体の識別を実現することができ、また、例えば、動画像処理のように時間的な制約のある時間内などで、優先順位に見合った識別結果を出すことが可能である。
【００９８】
また、本実施例に係る画像処理システムでは、２つ以上の映像入力装置を平行に設置して画像処理を行うことにより、処理対象とする物体までの距離を計算する。
従って、本実施例に係る画像処理システムでは、２つ以上のカメラなどを用いて、簡単な処理により、注目して対象としている物体までの距離を概算などすることができる。
【００９９】
以上のように、本実施例に係る画像処理システムでは、画像処理に際して処理量を比較的に低減させることができることから、例えば、バッテリ稼動が可能な小型の演算処理装置によっても処理が可能となり、このため、組み込みシステムやロボットに搭載することも可能となる。
【０１００】
ここで、本発明の構成としては、必ずしも以上に示したものに限られず、種々な構成が用いられてもよい。なお、本発明は、例えば本発明に係る処理を実現するためのプログラムなどとして提供することも可能である。
また、本発明の適用分野としては、必ずしも以上に示したものに限られず、本発明は、種々な分野に適用することが可能なものである。
【０１０１】
また、本発明において行われる各種の処理としては、例えばプロセッサやメモリ等を備えたハードウエア資源においてプロセッサがＲＯＭ（Ｒｅａｄ　Ｏｎｌｙ　Ｍｅｍｏｒｙ）に格納された制御プログラムを実行することにより制御される構成が用いられてもよく、また、例えば当該処理を実行するための各機能手段が独立したハードウエア回路として構成されてもよい。
また、本発明は上記の制御プログラムを格納したフロッピー（登録商標）ディスクやＣＤ（Ｃｏｍｐａｃｔ　Ｄｉｓｃ）−ＲＯＭ等のコンピュータにより読み取り可能な記録媒体や当該プログラム（自体）として把握することもでき、当該制御プログラムを記録媒体からコンピュータに入力してプロセッサに実行させることにより、本発明に係る処理を遂行させることができる。
【０１０２】
【発明の効果】
以上説明したように、本発明に係る画像処理装置などによると、例えば、画像中で注目したい対象を中心として、段階的に解像度を高めるとともに画像範囲を小さくして処理対象となる画像に基づく画像情報を取得し、取得した画像情報に基づく処理を行うようにしたため、画像中で注目したい対象を的確に捉えることができ、例えば、リアルタイムでの柔軟で適応範囲の広い画像処理が可能となり、特に、動画像における物体の識別を効率的に行うことが可能となる。
【図面の簡単な説明】
【図１】本発明の一実施例に係る画像処理システムのハードウエアの構成例を示す図である。
【図２】記憶装置内のソフトウエアの構成例を示す図である。
【図３】本発明の一実施例に係る画像処理システムの構成例を示す図である。
【図４】リサンプリング処理の様子の一例を示す図である。
【図５】リサンプリングする起点を変更する順番の一例を示す図である。
【図６】階層型のリサンプリング処理の様子の一例を示す図である。
【符号の説明】
１・・計算機、　２・・入力装置、　３・・出力装置、　４・・演算補助装置、
５・・画像入力装置、　１１・・記憶装置、　１２・・演算装置、
２１・・画像データ、　２２・・画像リサンプリング機構、
２３・・段階的画像処理機構、　２４・・制御機構、
３１・・２次元映像センサ、　３２・・映像キャプチャー装置、
３３・・ＤＭＡ制御装置、　３４・・メモリ、　３５・・演算処理装置、
４１・・プログラム、　４２・・ＤＳＰユニット、　５１・・画素の構成、
５２、６１・・入力画像、　５３〜５５、６２〜６５・・リサンプリング画像、[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus and an image processing method for processing an image, and more particularly, to an image processing apparatus and an image processing method capable of efficiently identifying an object in a moving image.
[0002]
[Prior art]
In conventional image processing, for example, linear processing is performed based on two-dimensional data obtained by quantizing a video (image) projected on a two-dimensional plane at sampling points at fixed intervals. In such processing, there is a problem that the amount of processing data is large and real-time image processing is very difficult.
[0003]
In the method of performing processing by software using a high-speed CPU (Central Processing Unit), there is a problem that a processing device is expensive, large and heavy, and power consumption is large.
In addition, devices called artificial retinal devices have been developed in which image processing is implemented in hardware, but because of the hardware, it is difficult to modify the processing algorithm or adapt to changes, or for individual purposes. Had to be rebuilt.
[0004]
Also, in the human eye, the processing volume is reduced by using so-called foveated vision, which increases the resolution in the center of the visual field and lowers the resolution in the surrounding area, and enables processing of a wide range of visual fields by eye movement. I have.
In this regard, the “Development of a Visual Information Processing Processor for the Brain-Type Information Processing System” (Science and Technology Promotion Agency, Strategic Basic Research Promotion Project (CREST), “Create the Brain” Research Area Results) Although an artificial retinal chip has been developed, there is a problem that it is difficult to reduce the cost because the application is not versatile.
[0005]
Conventionally, a visual system close to the human fovea to which a logarithmic scale of a polar coordinate system is applied has been devised (for example, see Non-Patent Document 1). However, in such a visual system, there is a problem in that a computer and an algorithm based on a normal orthogonal coordinate system require cost for coordinate transformation and inverse transformation.
[0006]
Conventionally, an imaging system for providing a panorama like a human eye by providing a mirror for converting into a fovea shape has been devised (for example, see Patent Document 1). However, such a system is expensive due to the need for additional special optics, and changes the field of view corresponding to human eye movement to cover a low resolution area. However, there is a problem that it is necessary to add a device for this purpose.
[0007]
In addition, Japanese Patent Application No. 2001-261899, entitled "Image Processing Apparatus and Method", discloses a method of processing using software, in which the image is resampled into a foveated shape and then processed. . In this method, linearity information is lost because a non-linear transformation is performed.
[0008]
[Non-patent document 1]
"Visual System Using Foveal Vision Sensor", "Transactions of the Institute of Electronics, Information and Communication Engineers", January 1994, J77-D-II, No. 1, p. 119-130
[Patent Document 1]
JP 2000-341568 A
[0009]
[Problems to be solved by the invention]
However, although various image processing technologies have been developed as shown in the above-mentioned conventional example, the technology for accurately capturing a target to be noticed in an image is still insufficient, and further development is desired. Was. Specifically, in the related art, for example, it is difficult to perform image processing in a real-time flexible and adaptable range, and particularly, it is difficult to identify an object in a moving image.
[0010]
The present invention has been made in view of such conventional circumstances, and has as its object to provide an image processing apparatus and an image processing method that can accurately capture a target of interest in an image. More specifically, the present invention enables, for example, flexible and wide-ranging image processing in real time, and particularly enables efficient identification of an object in a moving image.
[0011]
[Means for Solving the Problems]
In order to achieve the above object, an image processing apparatus according to the present invention processes an image as follows.
That is, the image information acquiring unit acquires the image information based on the image to be processed by increasing the resolution stepwise and reducing the image range. Then, the image information processing means performs a process based on the image information obtained by the image information obtaining means.
[0012]
Therefore, by performing the process of gradually increasing the resolution and reducing the image range, it is possible to accurately capture a target of interest in the image. In addition, for example, flexible and wide-range image processing can be performed in real time, and in particular, it is possible to efficiently identify an object in a moving image.
[0013]
Various images may be used as the image to be processed. For example, a still image may be used, or a moving image may be used.
In addition, various methods may be used as a method of increasing the resolution stepwise or a method of gradually reducing the image range.
[0014]
The resolution corresponds to, for example, the degree to which an image looks fine. For example, the same resolution as that of the original image to be processed or a lower resolution is used.
Further, the image range corresponds to, for example, a range of an image constituted by the acquired image information. For example, the same range as the original image to be processed or a smaller range is used.
[0015]
Various kinds of information may be used as the image information. For example, part or all of the information of the image to be processed may be used as it is, or part of the information of the image to be processed or All can be processed and used.
In the image information processing means, various processes may be performed.For example, the target of interest is located at the center of the image range, and the position of the target can be accurately determined based on image information having a relatively high resolution. As in the process of detecting, it is possible to perform a process of obtaining various pieces of information on a target of interest.
[0016]
Further, in the image processing apparatus according to the present invention, as one configuration example, in the image information acquisition unit, the initial stage image information acquisition unit acquires image information based on an image to be processed at a resolution and an image range in an initial stage. Then, the stepwise image information obtaining means changes the resolution and the image range in the initial stage to a higher resolution in a stepwise manner and changes to a smaller image range to obtain image information based on the image to be processed. Further, the image information processing means performs processing based on the image information acquired at least in the final stage.
[0017]
Various resolutions and image ranges may be used in the initial stage. For example, a relatively low resolution and a relatively large image range are used.
Further, since the image information acquired at the final stage has the highest resolution, the image information processing means mainly performs processing based on the image information acquired at the final stage, for example.
[0018]
In the image processing device according to the present invention, as one configuration example, pixel information is used as image information.
For example, if the image to be processed is composed of digital pixel information, the pixel information can be used.For example, if the image to be processed is an analog image, the analog image can be used. The image can be converted into an image composed of digital pixel information and the pixel information can be used.
[0019]
Further, in the image processing apparatus according to the present invention, as one configuration example, in the image information acquisition unit, the reference pixel information identification unit uses the pixel information to be a reference in the next stage based on the pixel information acquired in a predetermined stage. Identify. Then, the image information obtaining means sets an image range in the next step based on the pixel information specified by the reference pixel information specifying means based on the pixel information obtained in the predetermined step.
[0020]
Therefore, when the resolution is increased stepwise to reduce the image range, the pixel information specified in the (n + 1) -th step is determined based on the pixel information specified in the n-th step (where n is an integer of 1 or more). Thus, for example, it is possible to perform a process that focuses on a target corresponding to the reference pixel information.
[0021]
Note that various methods may be used to specify the reference pixel information in the next stage based on the pixel information acquired in the predetermined stage. For example, the pixel information acquired in the predetermined stage may be used. A method of specifying reference pixel information in the next step from the information can be used.
In addition, as the criterion relating to the pixel information as a criterion, a criterion for various items may be used.For example, a criterion for determining the center of an image range can be used. Pixel information positioned at or near the center of the image range can be specified as reference pixel information.
[0022]
Various methods may be used to specify the reference pixel information. For example, it is possible to use a method of recognizing and specifying preset target pixel information based on its characteristics. More specifically, the target pixel information can be recognized and specified based on features such as colors and outlines different from the background. Note that, for example, it is also possible to adopt a configuration in which reference pixel information is specified based on an instruction from a person (user).
[0023]
In the image processing device according to the present invention, as one configuration example, the image information acquisition unit controls the resolution by controlling the interval at which pixel information is thinned.
Note that various modes may be used as a mode for controlling the interval for thinning out pixel information.
A case where pixel information is not thinned out may be used.
Normally, the smaller the interval for thinning out the pixel information, the higher the density of the pixel information is obtained. Therefore, the higher the interval, the lower the resolution.
[0024]
Further, in the image processing device according to the present invention, as one configuration example, the image information acquisition unit controls the resolution by controlling a mode of averaging the pixel information.
Note that various modes may be used as a mode for controlling the mode of averaging the pixel information.
A case where pixel information is not averaged may be used.
Normally, the smaller the number of pieces of pixel information to be averaged, the higher the resolution, and the larger the number, the lower the resolution.
[0025]
Further, in the image processing apparatus according to the present invention, as one configuration example, the image information acquisition unit acquires pixel information corresponding to a fixed-length two-dimensional array at each stage.
Therefore, the number of pieces of pixel information acquired in each stage can be made the same, and the memory for storing the acquired pixel information and the configuration for processing the pixel information can be simplified.
[0026]
Various types of fixed-length two-dimensional arrays may be used. For example, the number of pixels in the horizontal axis direction of the rectangular coordinate system is fixed and the number of pixels in the vertical axis direction is fixed. Things are used.
[0027]
Further, in the image processing apparatus according to the present invention, as one configuration example, the image information acquisition unit may include, in each of the steps, a power of 2 by a number m to be reduced by one as the step proceeds by one. ^m The pixel information is acquired at a pixel interval corresponding to the number of pixels.
By using a power of two in this manner, the processing can be simplified.
[0028]
Further, in the image processing apparatus according to the present invention, as one configuration example, the image information acquisition unit acquires the pixel information in a plurality of types of pixel information acquisition modes in the same stage, and outputs any type of pixel in the same stage. Based on the pixel information acquired by the information acquisition mode, the reference pixel information is specified in the next stage.
Therefore, for example, even when it is difficult to specify the reference pixel information with the pixel information acquired by one type of pixel information acquisition mode, the pixel information is acquired by the plural types of pixel information acquisition modes. , The reference pixel information can be specified.
[0029]
Note that various modes may be used as a plurality of types of pixel information obtaining modes.
Various numbers may be used as the number of the plurality of types of pixel information acquisition modes.
In addition, the process of switching the pixel information acquisition mode at the same stage may be performed at any stage, and it is particularly effective to perform the process at the initial stage or at a stage close to the initial stage.
[0030]
Further, in the image processing apparatus according to the present invention, as one configuration example, the image information acquisition unit sequentially switches a plurality of types of pixel information acquisition modes in which all pixel information is acquired in one round in one round, Get pixel information.
Therefore, when the switching of a plurality of types of pixel information acquisition modes makes one round in the same stage, all the pixel information in the stage is acquired, so that the process of specifying the reference pixel information can be made more efficient.
[0031]
Further, in the image processing apparatus according to the present invention, as one configuration example, the image information acquisition unit specifies a plurality of pieces of reference pixel information based on the pixel information acquired in the same stage.
Therefore, it is possible to specify a plurality of pieces of reference pixel information and to perform, for example, a process based on each of the plurality of pieces of reference pixel information.
In addition, various numbers may be used as the number of pieces of pixel information serving as a plurality of references.
[0032]
Further, in the image processing apparatus according to the present invention, as one configuration example, the image information acquisition unit associates a plurality of pieces of reference pixel information with the processing priority based on the pixel information acquired in the same stage. Identify.
Therefore, since the plurality of pieces of reference pixel information are specified in association with the processing priority, for example, the processing based on each of the plurality of pieces of reference pixel information is sequentially performed according to the priority. Can be.
[0033]
Note that various orders may be used as the order of assigning priorities when specifying a plurality of pieces of reference pixel information. For example, the first set of pixel information of a target to be noted The priority is assigned, and the second and subsequent priorities are sequentially assigned to the other pixel information, or the priority is assigned in order from a target whose distance to the image processing apparatus is short. Such an embodiment can be used.
[0034]
Further, in the image processing apparatus according to the present invention, as one configuration example, a plurality of images including the same target image are used as images to be processed. Then, in the image information processing means, the target distance detection means performs a process of detecting a distance to the target based on a plurality of images including an image of the same target.
Therefore, the distance to the target can be detected based on the plurality of images.
[0035]
The distance from the object to the object may be used as the distance to the object. For example, the distance between the camera provided in the image processing apparatus and the object is used.
In addition, an accurate distance is not necessarily used as the distance to the target, and for example, an approximate distance may be used.
[0036]
Further, in the image processing apparatus according to the present invention, as one configuration example, in the image information obtaining unit, the distance detection target image information specifying unit specifies the target image information whose distance is to be detected by the target distance detection unit.
As a target for detecting the distance, for example, a target corresponding to the above-described reference pixel information may be used. In this case, by specifying the reference pixel information, the reference pixel information is determined. Can be specified as image information of a target whose distance is to be detected.
[0037]
In addition, various methods may be used to specify the target image information in the image. For example, the target image information may be specified in advance by recognizing the target image information in the image based on its characteristics. A method can be used, and specifically, the target image information can be recognized and specified in the image based on features such as colors and outlines different from the background. Further, for example, it is also possible to adopt a configuration in which target image information is specified in an image based on an instruction from a person (user).
[0038]
In the image processing apparatus according to the present invention, as one configuration example, the image input unit inputs an image to be processed.
As the image input means, various means may be used. For example, an image pickup device such as a camera may be used, or a means for inputting image information from an external device or an external memory. Can also be used.
Further, various numbers may be used as the number of image input means.
[0039]
Further, in the image processing apparatus according to the present invention, as one configuration example, a moving image is used as an image to be processed.
Further, in the image processing method according to the present invention, an image is processed as follows.
That is, image information based on the image to be processed is acquired by increasing the resolution stepwise and reducing the image range, and processing based on the acquired image information is performed.
[0040]
As a specific application example of the present invention as described above, a function according to the present invention can be added to a camera mounted on a device (moving body) such as a moving robot. The present invention relates to a process for recognizing and following an object of interest, such as a ball or a goal, in such a moving object, for example, by recognizing and avoiding an object other than the object of interest as an obstacle. The process of recognizing other objects together with the target to be given attention and assigning priority to each object, the process of detecting the exact position of the target of interest, the process of detecting the distance to the target of interest, etc. It can be helpful to perform. Further, in the moving body to which the present invention is applied, for example, since the load of image processing is relatively light and the processing speed can be increased, it is possible to process a moving image captured while moving in real time. .
[0041]
BEST MODE FOR CARRYING OUT THE INVENTION
An embodiment according to the present invention will be described with reference to the drawings.
First, an outline of an image processing system to which an image processing apparatus and an image processing method according to the present invention are applied will be described.
FIG. 1 shows a computer 1 having a storage device 11 and a calculation device 12, an input device 2, an output device 3, a calculation assistance device 4, and a hardware configuration example of an image processing system to which the present invention is applied. The image input device 5 is shown.
[0042]
The image processing system shown in FIG. 1 operates with a computer 1 including a storage device 11, an arithmetic device 12 for executing a program based on the storage device 11, and input / output devices (input device 2 and output device 3) as a host. Further, a calculation assisting device 4 suitable for processing a large amount of information may be connected. Further, as one of the input devices 2, an image input device 5 having a function of capturing a moving image, for example, is provided. Note that a plurality of arithmetic devices 12 may be provided, and as an example, each of the arithmetic devices is assigned a role according to a load, such as an arithmetic device for processing input information and an arithmetic device for processing output information. It is also possible to have
[0043]
FIG. 2 shows a configuration example of software inside the storage device 11. The program and data for the arithmetic device 12 stored in the storage device 11 have the configuration shown in FIG. The image data 21 to be processed is processed by a program. The program has an image resampling mechanism 22, a stepwise image processing mechanism 23, and a control mechanism 24.
[0044]
The image resampling mechanism 22 has a function of resampling video (image) data sampled at equal intervals input from the image input device 5 around, for example, a point to be watched (gazing point). The stepwise image processing mechanism 23 has a function of processing the resampled image. The control mechanism 24 has a function of judging a situation based on the abstracted information from the stepwise image processing mechanism 23 and determining an action. The control mechanism 24 may have, for example, a function of outputting a control signal to an external device via the output device 3 in addition to a function of changing or calculating an internal state. Can be controlled.
[0045]
By the way, in the human retina, the density of pixels is highest around the center of the visual field, and becomes sparser toward the periphery. In addition, when there is an object to be focused on, by adjusting the line of sight to the object by eye movement, an image around the object to be focused can be intensively captured by the retina. This has the effect of reducing the overall processing amount by focusing on image processing around the object of interest.
[0046]
On the other hand, in a general imaging device, each pixel is usually at equal intervals, and the pixels are not arranged intensively so as to correspond to a human retina. Also, it is very difficult to prepare a mechanism for instantaneously and precisely changing the direction of the camera vertically and horizontally, and even if it can be prepared, it will be very expensive.
[0047]
Accordingly, in the present embodiment, for example, based on an input image imaged at a regular pixel arrangement by an image sensor similar to the conventional one, first, pixel information is reacquired (resampling) by thinning out the image to obtain the resolution. By using a low image to determine the area (image range) to be processed centrally, and then increasing the resolution step by step, the processing of unnecessary image areas can be greatly reduced. Accordingly, it is possible to reduce the overall processing amount and shorten the overall processing time.
[0048]
In addition, the image data used to obtain the region to be centrally processed is not necessarily limited to the data of the image obtained by thinning out the input image. It is also possible to use image data output by hardware or a digital signal processor (DSP: Digital Signal Processor) or the like.
[0049]
Further, in the configuration of the present embodiment as described above, by setting an area in which an object to be centrally processed exists as an image area for increasing the resolution, a mechanism similar to eye movement can be easily realized, Inexpensive and high-speed viewpoint movement can be made possible. Further, in the configuration of the present embodiment, since the resolution can be maintained high near the center of the processing, the overall processing amount can be reduced without lowering the recognition rate near the center of the processing. It is possible.
[0050]
It is said that the eyeball of an animal vibrates little by little and performs information processing with a resolution higher than the interval between nerve cells in the retina.
Therefore, in the present embodiment, for example, in the case where the center point to be intensively processed cannot be found due to the low resolution in the primary resampled image, the starting point for resampling is shifted with the low resolution. By doing so, it becomes possible to treat all pixels as processing targets without causing overlapping of pixels to be processed. If the center point of the process is found, more reliable identification is performed using a higher-resolution image in which the resampling interval is narrowed around the center point of the process.
[0051]
In the present embodiment, by moving the point to be watched (gaze point) in the resampling process using a process corresponding to a motion similar to the eyeball motion of the animal, thinning is performed according to the resampling interval. Information that has been lost can be processed with a time lag, which can increase the substantial resolution.
[0052]
Next, more detailed examples according to the present invention will be described.
First, a first embodiment will be described.
FIG. 3 shows a configuration example of an image processing system according to an embodiment of the present invention. The image processing system shown in FIG. 1 includes a two-dimensional image sensor 31, an image capture device 32, a DMA (Direct Memory Access) control device 33, a memory 34 storing a program 41, and a DSP (Digital Signal Processing). An arithmetic processing unit 35 having a unit 42 is provided.
[0053]
The two-dimensional image sensor 31 is an image input device configured using, for example, a CCD (Charge Coupled Device) with a lens or a CMOS (Complementary Metal-Oxide Semiconductor), and detects, for example, a moving image (moving image). The information of the video is sequentially transmitted to the video capture device 32 in the YCrCb format. The two-dimensional image sensor 31 corresponds to, for example, the image input device 5 shown in FIG.
[0054]
In this example, as the pixel value information of one pixel, the number of bits of information of Y (stimulus value) is 8 bits, the number of bits of information of Cr (red) is 4 bits, and Cb (blue) The information in which the number of bits of the information is 4 bits is used. In addition, from the two-dimensional image sensor 31 to the image capture device 32, Y information is output in units of 8 bits, Cr information is output in units of 4 bits, and Cb information is output in units of 4 bits. You.
[0055]
The video capture device 32 captures YCrCb pixel values for each dot clock (clock of one pixel at a time) of video information input from the two-dimensional video sensor 31, and transfers the captured pixel values to the DMA control device at regular intervals. The data is DMA-transferred to the memory 34 of the arithmetic processing unit 35 via 33. Information specifying the address of the memory 34 is output from the video capture device 32 to the DMA control device 33 in units of 16 bits, and the data of the captured pixel value is output in units of 16 bits (= 8 + 4 + 4). .
[0056]
A program 41 is stored in another area on the memory 34 (an area other than an area for storing pixel value data), and the arithmetic processing unit 35 performs processing according to a procedure defined by the program.
Further, the arithmetic processing unit 35 is provided with a DSP unit 42, and the DSP unit 42 also functions as an arithmetic auxiliary device. The arithmetic assist device corresponds to, for example, the arithmetic assist device 4 shown in FIG.
[0057]
The procedure of the process performed by the DSP unit 42 is embedded in the program 41 described above.
Further, the arithmetic processing unit 35 outputs a DMA control signal for controlling the DMA transfer to the DMA control unit 33.
Note that, between the arithmetic processing unit 35 and the memory 34, information on the address of the memory 34 is input / output in units of 32 bits, and pixel value data is input / output in units of 32 bits.
[0058]
An example of an operation performed by the image processing system of the present example will be described.
FIG. 4 shows, as an example of a state of the resampling process performed by the image processing system of the present example, an input image 52, a pixel configuration (only a part) 51 of the input image 52, and resampled images 53 to 55. Is shown. In this example, a case where attention is paid to a circular object included in the input image 52 is shown.
[0059]
In a first-order resampling process, which is an initial stage, pixels of the original input image 52 in the horizontal axis direction of the orthogonal coordinate system are thinned out to reduce the number of pixels to と と も に and to reduce pixels in the vertical axis direction. Are thinned, and the pixel point that reduces the number of pixels to 1/4 is set as a resampling point, and an image (thinned image) of the entire range of the input image 52 is reduced to a first order by using a pixel value corresponding to the resampling point. It is acquired as a sampling image 53. When the horizontal axis direction and the vertical axis direction are combined, the number of pixels is thinned out to 1/16 (= (1/4) × (1/4)).
[0060]
Next, a circular object that is a target of interest included in the first-order resampled image 53 is identified, and any one of the pixels (for example, the center of the object or the vicinity of the center of the object, Pixel) is set as the center of the image range in the next stage, the second resampling process.
[0061]
Next, in the second resampling process, pixels in the horizontal axis direction of the orthogonal coordinate system are thinned out to reduce the number of pixels to と と も に and pixels in the vertical axis direction with respect to the original input image 52. Then, a pixel point where the number of pixels is reduced to 1/2 is set as a resampling point, and 1/4 (= 1/2 in the horizontal axis direction and 1 in the vertical axis direction) An image (thinned-out image) in the range of (/ 2) minutes is acquired as the second-order resampled image 54. In addition, when the horizontal axis direction and the vertical axis direction are combined, the number of pixels is thinned out to ４ (= (１ ／) × (１ ／)). The total number of pixels resampled by the primary resampling process is the same as the total number of pixels resampled by the secondary resampling.
[0062]
Next, a circular object that is a target of interest included in the second-order resampled image 54 is identified, and any of the pixels constituting the image of the object (for example, the center of the object or the vicinity of the center is predicted. Pixel) is set as the center of the image range in the next stage, the third resampling process.
[0063]
When specifying the pixel such as the center of the same target of interest, the pixel specified in the n-th (where n is an integer of 1 or more) next resampling process is directly used in the next (n + 1) -th order Can be specified by the resampling process described above, but it is preferable to perform the specification each time the resolution of the resampled image is increased as in this example because the specific accuracy can be increased.
[0064]
Next, in the third resampling process, a pixel point having the same resolution without thinning out pixels is set as a resampling point in the original input image 52, and a pixel value corresponding to the resampling point is calculated. Then, an image in a range of 1/16 (= ５２ in the horizontal axis direction and 1/4 in the vertical axis direction) of the input image 52 is acquired as the third-order resampled image 55. The total number of pixels resampled by the third resampling process is equal to the total number of pixels resampled by the first resampling process and the second resampling process.
[0065]
Here, such a resampling process will be described using mathematical expressions.
In this example, pixels are resampled using a calculation formula as shown in Expression 1.
[0066]
(Equation 1)

[0067]
Here, (Xi, Yj) represents the coordinate value of the XY orthogonal coordinate system, and represents the coordinate value of the pixel point to be resampled. L represents, for example, the number of resamplings in the X-axis direction, which is the horizontal axis direction, and M represents, for example, the resampling number in the Y-axis direction, which is the vertical axis direction. (X0, Y0) represents a coordinate value at which resampling starts, and d represents a resampling interval.
[0068]
When the width of the image area to be resampled in each stage is represented by X, and the height of the image region to be resampled in each stage is represented by Y, X> d × L and Y> d × M are obtained. is necessary.
In this example, the object to be identified by paying attention to the resampling start coordinate values (X0, Y0) used in the resampling process in the next stage of increasing the resolution from the resampling result in a certain stage and performing resampling in the next stage By setting the value at the center of the range, efficient processing is realized.
[0069]
Further, when the sampling interval d is a power of 2, calculation becomes easy. For example, in order to obtain the processing of the maximum resolution by the three-stage resampling processing as in this example, the resampling interval d1 = 2 in the first resampling processing stage. ² = 4, and the resampling interval d2 = 2 at the stage of the second resampling process ¹ = 2, and a resampling interval d3 = 2 at the stage of the third resampling process ⁰ = 1 can be used.
[0070]
Further, as in this example, if the number of resamplings in the horizontal axis direction (for example, L = X / d) and the number of resamplings in the vertical axis direction (for example, M = Y / d) are common values in each stage. The data area (the total number of pixels in the horizontal axis direction and the total number of pixels in the vertical axis direction) for performing image processing can be made constant at each stage, and the processing mechanism can be simplified.
[0071]
In the case of this example, at each stage, 1/16 (= (１ ／)) of the original input image 52 is used. ² ), And the processing can be completed with a data amount of 3/16, which is three times the value, in the total of the overall processing over three stages.
[0072]
Further, as described above, for the image after resampling, for example, normal image processing is performed to identify or discriminate the target of interest. As a method of performing such identification or determination, for example, a method of using a distribution histogram based on pixel values, a method of recognizing a shape from connection of edges after performing edge detection, or a neural network For example, a method of performing discrimination using such a method can be used.
[0073]
Next, a second embodiment will be described.
Note that the image processing system of the present embodiment has a configuration similar to that of the first embodiment shown in FIG. 3, and performs the same operation as that of the first embodiment.
In the first-order resampling process shown in FIG. 4 and the like, image data of an object to be focused on enters between resampling intervals (ie, is thinned out), and May be overlooked.
[0074]
Therefore, in the image processing system of the present example, when the coordinate value of the pixel point to be determined as the pixel point of the object of interest cannot be determined, the mode shifts to the search mode. Then, in the search mode, at the same stage, the gazing point is shifted at a minute interval for each resampling process, for example, thereby shifting the resampling start coordinate value (X0, Y0), thereby performing a plurality of resampling operations. Perform processing. In this way, at the same stage, missing portions of information due to a certain resampling process are processed at different times, thereby realizing interpolation of the missing portions.
[0075]
FIG. 5 shows an example of the order of the pixel points as starting points (resampling start coordinate values) at which resampling processing is performed at the same stage for 16 pixel points forming an image (part). In the example shown in the figure, the resampling process is performed in order of numbers 1, 2, 3,..., 14, 15, and 16 starting from the pixel point of each number.
[0076]
In this example, in particular, in the first-order resampling processing, since the resolution is low, there is a case where a center point to be focused and processed is not found. In such a case, by shifting the starting point for performing the resampling process in the order shown in the figure while maintaining the low resolution, all the pixels for the resampling process can be performed without overlapping. Can be processed.
Then, when a center point to be processed with attention is found, the process shifts to a resampling process in the next stage centering on the point, and thereby, based on a high-resolution image with a narrowed resampling interval, Perform reliable identification.
[0077]
As described above, in the image processing system of this example, by moving the gazing point by using a process corresponding to a motion similar to the eyeball motion of an animal, information on pixels that have been thinned out at resampling intervals is obtained. Can be processed with a time lag, which can increase the substantial resolution.
It should be noted that the overall processing over multiple stages is essentially processing important information at short time intervals (ie, processing for a long time) while processing non-critical information at long time intervals (ie, (Processing for a short time) is realized, and the total processing amount is reduced in a well-balanced manner.
[0078]
Further, for example, when continuously processing images of a plurality of frames connected in chronological order such as a moving image, a center point of interest in the image processing is obtained for each frame image, and a difference between different frames is determined. By calculating the difference between the coordinate values of the center point, it is possible to easily detect the movement of the object of interest and calculate the moving speed of the object of interest.
[0079]
Next, a third embodiment will be described.
Note that the image processing system of this example has a configuration similar to that of the first embodiment shown in FIG. 3, and is similar to that shown in the first embodiment or the second embodiment. Perform the operation.
[0080]
In the resampling process shown in FIG. 4, there may be two or more objects to be noted and identified in one image (one frame). In such a case, in the image processing system of the present example, for example, as shown in FIG. 4, after one object of interest is determined at the final stage, the other object of interest is determined. The secondary resampling process is performed by changing the location of the resampling process to the corresponding coordinate value, and the process of increasing the resolution from the secondary resampling process is performed. As described above, the other object of interest can be efficiently processed without redundant processing such as the primary resampling processing.
[0081]
FIG. 6 shows an example of the state of performing the resampling processing on a plurality of objects of interest hierarchically (hierarchical resampling processing), for example, the same as the input image 52 shown in FIG. An input image 61 that is an image and a resampled image that is the same image as the resampled images 53, 54, and 55 for the circular object (here, referred to as the first target of interest) shown in FIG. 62, 63, and 64, and a second-order resampled image 65 of an object (here, referred to as a second target of interest) composed of a combination of triangles and rectangles included in the input image 61. is there.
[0082]
In the example of FIG. 6 described above, as the first order processing, the stepwise resampling processing for the first target of interest is performed from the first resampling processing, and then the second order processing is performed as the second order processing. The stepwise resampling process for the second target is performed from the second resampling process. The second target of interest is specified as the target of interest together with the first target of interest, for example, in the first processing.
[0083]
Further, when a plurality of objects of interest are set in the image as described above, priorities are determined for each of the plurality of objects of interest, and the respective objects of interest are identified in the order of the priority. It can also be configured to be. In such a configuration, for example, even when the processing is terminated due to time out when the processing time is limited as in the case of moving image processing, an accurate position or the like is detected for a high priority object, and For an object with a low image density, an approximate position is detected, which is suitable for real-time moving image processing.
[0084]
Next, a fourth embodiment will be described.
Note that the image processing system of this example has the same configuration as that shown in FIG. 3 of the first embodiment, and is similar to that shown in the first to third embodiments. Perform the operation.
[0085]
In the image processing system of this example, two cameras are installed in parallel at an appropriate interval, and the input images of both cameras are, for example, the same as those shown in FIG. 4 of the first embodiment. Perform sampling processing.
For example, when the same object is gazed at the left and right input images obtained from the two cameras, if the target is an object at infinity, the same coordinate value is the gazing point in the left and right input images, but the object is close When, there is a difference in coordinate values between the gazing points in the left and right input images. Then, by using such a difference between the coordinate values, it is possible to calculate a distance (for example, a rough distance) from the target object.
[0086]
As a specific example, the distance between the center lines of the two left and right cameras is represented by D, the distance from a camera (for example, any one of the cameras) to the target object is represented by Z, and the target object is When the angle difference between two lines connecting the cameras with each other is represented by θ, and the difference between the coordinate values of the target object in the left and right cameras is approximated to be proportional to the angle difference θ, Equation 2 holds. Therefore, the distance Z from the camera to the target object is approximated as in Expression 3.
[0087]
(Equation 2)

[0088]
[Equation 3]

[0089]
However, in such approximation, the error increases as the target object moves away from the center of the input image.
Further, when detecting the distance to the target object in this way, for example, if a rotating pedestal on which a camera is mounted is prepared, not only can it be possible to widen the angle that can be processed, but also the target can be positioned at the center of the image sensor. The error can be reduced by adjusting the angle of the camera so that it is closer to the line.
[0090]
Also, for example, when processing a moving image, when the gazing points of the input images of both cameras are matched with an object that moves a large amount, the target object in the t (here, t is an integer of 1 or more) frame To determine the moving speed of the target object in the perspective direction by obtaining the distance Z (t) to the target object and the distance (Z (t + 1)) to the target object in a different frame such as the (t + 1) frame. Can be.
[0091]
As described above, in the image processing system according to the present embodiment, various processes are performed by resampling the video from the video input means in a stepwise manner and limiting the processing range in a stepwise manner. Further, the image processing system according to the present embodiment has a configuration in which pixel data to be resampled at each stage can be stored in a fixed-length two-dimensional array, and a power of 2 is used as an interval between pixels to be resampled. Is used.
[0092]
Therefore, in the image processing system according to the present embodiment, a rough place is first specified by a low-resolution resampled image, and then the resolution is gradually increased so as to focus on a place where an important object is likely to be present. As a result, the resampling process is performed, and the process of detecting the position of important objects is performed. It is possible to acquire accurate information on an object of interest based on the result of such resampling processing.
[0093]
Further, in the image processing system according to the present embodiment, for example, it is possible to realize processing corresponding to the retina or eye movement of a living organism using an optical device or an imaging device which is mass-produced and whose cost is reduced. Therefore, processing of a moving image or the like can be performed at low cost and at a high level. Further, in the image processing system according to the present embodiment, the resolution of the original image can be maintained near the pixel point of interest (for example, the gazing point). It is possible to reduce the overall processing amount without lowering the recognition rate or the like.
[0094]
Further, in the image processing system according to the present embodiment, when the center point of interest is not found in the resampling process in the initial stage, the pixel point serving as the starting point for performing the resampling process in the same stage. Is shifted, the information of the pixel value that would be lost in one type of resampling processing is totally interpolated by a plurality of types of resampling processing.
[0095]
Therefore, in the image processing system according to the present embodiment, for example, when the center point of the process of interest is not found, the resampling point is slightly shifted for each frame to change the pixel value of a certain resampling gap to another. , It is possible to specify the center point of the process of interest without increasing the resolution. As described above, in the image processing system according to the present embodiment, processing corresponding to eye movements of a living organism can be performed, and processing of moving images and the like can be performed at low cost and at a high level.
[0096]
Further, in the image processing system according to the present embodiment, a plurality of target objects are identified in one image by performing resampling processing and identification processing hierarchically on a plurality of target objects. Further, in the image processing system according to the present embodiment, the processing for such a plurality of targets is performed in the order of the priority assigned to each target.
[0097]
Therefore, in the image processing system according to the present embodiment, even when there are a plurality of objects to be identified in one image, the identification of the plurality of objects is realized by performing the hierarchical resampling process. In addition, for example, it is possible to output an identification result that matches the priority order, for example, within a time period with a time constraint such as moving image processing.
[0098]
In the image processing system according to the present embodiment, the distance to an object to be processed is calculated by performing image processing by installing two or more video input devices in parallel.
Therefore, in the image processing system according to the present embodiment, it is possible to roughly calculate the distance to the object of interest by simple processing using two or more cameras.
[0099]
As described above, in the image processing system according to the present embodiment, the processing amount can be relatively reduced during image processing. For example, processing can be performed by a small arithmetic processing device that can be operated by a battery. Therefore, it can be mounted on an embedded system or a robot.
[0100]
Here, the configuration of the present invention is not necessarily limited to the configuration described above, and various configurations may be used. Note that the present invention can be provided as, for example, a program for realizing the processing according to the present invention.
Further, the application field of the present invention is not necessarily limited to the above-described fields, and the present invention can be applied to various fields.
[0101]
Further, as various processes performed in the present invention, for example, a configuration in which a processor is controlled by executing a control program stored in a ROM (Read Only Memory) in a hardware resource including a processor and a memory is used. Alternatively, for example, each functional unit for executing the processing may be configured as an independent hardware circuit.
Further, the present invention can be understood as a computer-readable recording medium such as a floppy (registered trademark) disk or a CD (Compact Disc) -ROM storing the above-mentioned control program or the program (the program itself). The processing according to the present invention can be performed by inputting the program from the recording medium to the computer and causing the processor to execute the program.
[0102]
【The invention's effect】
As described above, according to the image processing apparatus and the like according to the present invention, for example, an image based on an image to be processed by gradually increasing the resolution and reducing the image range, centering on the target of interest in the image Acquiring information and performing processing based on the acquired image information, it is possible to accurately capture the target of interest in the image, for example, it is possible to perform flexible, wide-ranging image processing in real time, especially In addition, it is possible to efficiently identify an object in a moving image.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing system according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a configuration example of software in a storage device.
FIG. 3 is a diagram illustrating a configuration example of an image processing system according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating an example of a state of a resampling process.
FIG. 5 is a diagram illustrating an example of an order in which a starting point for resampling is changed.
FIG. 6 is a diagram illustrating an example of a state of a hierarchical resampling process.
[Explanation of symbols]
1 ··· Calculator, 2 ··· Input device, 3 ··· Output device, 4 ··· Auxiliary device,
5 image input device, 11 storage device, 12 arithmetic device,
21 image data, 22 image resampling mechanism,
23 · · · step-wise image processing mechanism, 24 · · · control mechanism,
31 2D image sensor 32 Image capture device
33 DMA controller, 34 memory, 35 arithmetic processing unit
41 .. Program, 42 .. DSP unit, 51 .. Pixel configuration,
52, 61 ... input image, 53-55, 62-65 ... resampling image,

Claims (17)

In an image processing apparatus for processing an image,
Image information obtaining means for obtaining image information based on an image to be processed by gradually increasing the resolution and reducing the image range,
Image information processing means for performing processing based on image information obtained by the image information obtaining means,
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The image information obtaining means includes: an initial stage image information obtaining means for obtaining image information based on an image to be processed at a resolution and an image range in an initial stage; Stepwise image information obtaining means for obtaining image information based on the image to be processed by changing to a small image range with changing,
The image information processing means performs processing based on at least the image information acquired in the final stage.
An image processing apparatus comprising: In the image processing device according to claim 1 or 2,
Pixel information is used as image information.
An image processing apparatus comprising: The image processing device according to claim 3,
The image information obtaining means has reference pixel information specifying means for specifying pixel information to be a reference in the next step based on the pixel information obtained in the predetermined step, and based on the pixel information obtained in the predetermined step. Setting the image range in the next stage based on the pixel information specified by the reference pixel information specifying means,
An image processing apparatus comprising: In the image processing device according to claim 3 or 4,
The image information acquisition unit controls the resolution by controlling the interval for thinning out the pixel information,
An image processing apparatus comprising: In the image processing device according to claim 3 or 4,
The image information obtaining means controls the resolution by controlling the mode of averaging the pixel information,
An image processing apparatus comprising: The image processing apparatus according to claim 5,
The image information obtaining means obtains, at each stage, pixel information corresponding to the fixed-length two-dimensional array.
An image processing apparatus comprising: The image processing apparatus according to claim 7,
The image information obtaining means obtains, at each stage, pixel information at a pixel interval corresponding to a power of 2 by a number to be decreased by one as the stage advances by one.
An image processing apparatus comprising: The image processing apparatus according to any one of claims 3 to 8,
The image information obtaining means obtains pixel information by a plurality of types of pixel information obtaining modes at the same stage, and performs the following step based on the pixel information obtained by any type of pixel information obtaining modes at the same stage. Identify pixel information as a reference,
An image processing apparatus comprising: The image processing apparatus according to claim 9,
The image information acquisition unit acquires pixel information by sequentially switching a plurality of types of pixel information acquisition modes in which all pixel information is acquired in one round in the same stage.
An image processing apparatus comprising: The image processing apparatus according to any one of claims 3 to 10,
The image information obtaining means specifies a plurality of reference pixel information based on the pixel information obtained in the same stage,
An image processing apparatus comprising: The image processing apparatus according to claim 11,
The image information obtaining means specifies a plurality of pieces of reference pixel information based on the pixel information obtained at the same stage in association with the processing priority.
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein
As an image to be processed, a plurality of images including an image of the same target are used,
The image information processing means has a target distance detection means for performing a process of detecting a distance to the target based on a plurality of images including an image of the same target,
An image processing apparatus comprising: The image processing apparatus according to claim 13,
The image information acquisition unit has a distance detection target image information specifying unit that specifies image information of a target whose distance is to be detected by the target distance detection unit.
An image processing apparatus comprising: The image processing apparatus according to any one of claims 1 to 14,
An image input unit for inputting an image to be processed;
An image processing apparatus comprising: The image processing apparatus according to any one of claims 1 to 15,
A moving image is used as an image to be processed.
An image processing apparatus comprising: In an image processing method for processing an image,
Acquire image information based on the image to be processed by increasing the resolution stepwise and reducing the image range, and perform processing based on the acquired image information.
An image processing method comprising:

Images