JPH08329033A - Method and device for image information processing, and its control method - Google Patents

Abstract

PURPOSE: To obtain necessary image information fast by detecting feature points of an inputted image, calculating a visual information quantity according to the positions of the feature points, and controlling an image input part so that the calculated visual information quantity increases. CONSTITUTION: A two-dimensional image inputted from a lens system 1 is made discrete through the sampling of an array sensor 2 and mapped in a multiple resolution space by a two-dimensional filter 3. The mapped image is converted by a conversion and encoding part 5 into a local pattern regarding the feature points of the input image detected by a feature detection part 4 and quantized by a quantization part 6 into pairs of position coordinates of the feature points and code words of the local patterns, so that they are inputted to respective cells of a probability automaton 7. The probability automaton 7 calculates mutual information quantities between different cells, and the position coordinates and visual information quantity of the feature points, and inputs the calculation results to an input parameter control part 8 to control the optical axis of the lens system 1 so that the visual information quantities become maximum.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention appropriately processes visual information to, for example, an input device, an image encoding / decoding device, an image recognition device, an image restoration device, a monitoring device, an autonomous vehicle, or an autonomous work robot. The present invention relates to an image information processing method and an apparatus therefor, and a control method thereof.

[0002]

2. Description of the Related Art Living organisms have a function of accurately recognizing an environment surrounding them by a finite number of processing devices and coping with it. The dynamic range of the signal required to recognize the environment is very wide assuming all situations. Considering visual information as an example, the number of visual sensors possessed by living organisms is of course limited, but the environment spreads in all directions. Therefore, in order to recognize the environment surrounding a creature without moving means, it is necessary to input a signal with the required resolution in all directions. However, if the organism has means of movement, that is, means for changing the observation parameter in the sensor, the load on the visual recognition system of the organism will be much lighter. This is because only the place that seems to be important for recognition is input with sufficient resolution for recognition, and if not, there is no need to input it.

A conventional image input device is a device for uniformly sampling a target image as seen in a CCD camera, a scanner or the like. What is obtained from such an image input device is image data of a finite area obtained at a certain resolution. Considering the image as a part of the visual information of the environment, it is an essential problem of visual information processing to estimate the three-dimensional visual information from the thus obtained two-dimensional image. The following two approaches have been taken to address such problems.

First of all, most of the researches on the visual system of living beings actively conducted in the 1980s, most of the researches by the mathematical model are the ideas of Marr (D. Marr: "Vision" WHFreem
It is no exaggeration to say that it originated in an and Co. NY (1982)). These studies are called visual calculation theory, and they have been developed by applying the theory of Markov random fields, line processes, renormalization transformation groups, and ideas of statistical physics. However, in these discussions, the visual information covers only what is already given as a finite number of image data, and deals with estimating the three-dimensional structure from the two-dimensional image set. This corresponds to, for example, looking at a picture or a picture and estimating its three-dimensional world. If we try to estimate a three-dimensional structure using only the information we have, the problem will generally be a poor setting in the sense that the solution will be indeterminate, so they are dealing with it by using their knowledge.

On the other hand, at the same time, a method was proposed in which information sufficient for recognition was prepared by controlling the visual input system itself, and then the environment was recognized. Ball
Animate Vision by ard (DH Ballard: "Behav
ioural constraints on animate vision ", image and vi
sion computing, Vol.7, No.1, pp.3-9 (1989)). This methodology is to eliminate the defective setting property existing in the initially inputted visual information by using the input data by another observation parameter. The observation parameters include the optical axis direction of the optical system, zooming, etc., but the most important thing in this technology is to determine "what to look for next" and "where to observe next". This is a method of controlling observation parameters.

1. Method by Ballard et al. (DH Ballard
and CM Brown: "Principles of Animate Vision", CVG
IP: IMAGE UNDERSTANDING, Vol.156, No.1, pp.3-21 (Au
g.1992)) In the visual environment recognition system constructed by Ballard et al., the image input device uses a foveal vision that samples a narrow area near the optical axis at high resolution and a low resolution away from the optical axis. Peripheral observation (per
ipheral vision). It is said that an object can be recognized if it is caught by foeval vision. Knowledge data is represented by a tree structure such as an IS-A tree or a part-of tree, and a probability structure is introduced into the relationship between objects. Based on this tree structure and the probability structure, an effective function (utility function) of the operation is defined by the amount of information obtained when performing a certain operation and the energy consumed for it, and using this utility function, the following It adopts a strategy of determining behavior. With such a strategy, an environment can be recognized in a shorter time.

2. In the system of Ballard et al. Mentioned above, a method of directly searching for an object to be searched next was adopted.
Wixson et al. Have proposed an indirect search method as an observation point control method for searching for a target object (LE Wixon an
d DH. Ballard: "Using intermediate objects to impr
ove the efficiency of visual search ", Int'l. J. Com
puter Vision, 12: 2/3, pp.209-230 (1994)). In the indirect search method, the search is performed by the spatial positional relationship between the object identified by observation and the target object. For example,
If the target object is a coffee cup and the identified objects are a desk, a chair, and a blackboard, the input system is controlled so that the position of the desk that has the strongest spatial relationship with the coffee cup is observed with higher resolution. It

In addition, systems such as Brooks (RA Brooks:
"New Approaches to Robotics", Science, Vol.25, pp.12
27-1232 (1991)) has several basic processing programs that connect sensor inputs and actuator outputs. Tani et al. Have proposed a system in which a rule existing in a sensor input as a time-series signal vector is acquired by learning and the rule is used for an action plan (JP-A-6-27).
4224). According to this method, a system adapted to an unknown environment can be constructed. It also provides a mechanism to select one of the possible actions, even if there are multiple possible actions.

The conventional representative theory has been introduced above.
Other proposals are as follows.

R. Rimey and CM Brown: "Task-Oriente
d Vision with Multiple Bayes Nets ", in" Active Vis
ion ", A. Blake and A. Yuille (Eds.) MIT press (1992) S. Geman and D. Geman:" Stochastic Relaxation, Gib
bs Distributions, and the Bayesian Restoration of
Image ", IEEE Trans. On Pattern Anal. Machine Intel
l., Vol.6, No.6, pp721-741 (Nov.1984) B. Gidas: "A Renormalization Group Approach to Ima
ge Processing Problems ", IEEE Trans. on Pattern An
al. Machine Intell., Vol.11, No.2, pp.164-180 (Feb.
1989) Kawato and Inui: "Computational Theory of the Visu
al Cortical Areas ", IEICE Trans., Vol.J73-D-II, No.
8, pp.1111-1121 (Aug.1990) DV Lindley: "On a measure of the infomation prov
idedby an experiment ", Ann. Math. Stat., vol.27, pp.
986-1005 (1956) KJ Bradshaw, PF McLauchlan, ID Reid and DW
Murray: "Saccade and pursuit on an active head / ey
e platform ", Image and Vision Computing, Vol.12, n
o.3, pp.155-163 (Apr.1994) JG Lee and H. Chung: "Global path planning for m
obile robot with grid-type world model ", Robotics
and Computer-Integrated Manufacturing, Vol.11, no.
1, pp.13-21 (1994)

[0011]

However, most of the above calculation theories are based on (a set of) given images.
Since we are discussing the information that can only be obtained from the above, the results obtained are estimates only. Since the world is described using the observer-centered coordinate system, handling of moving objects is complicated.

On the other hand, in Animate Vision, since the world is described using the object center coordinate system, handling of moving objects and the like becomes relatively simple. At that time, however, the most important observation point control is There are some problems below.

1. We have not discussed the method of recognizing the smallest unit of the object that constitutes knowledge. That is, the discussion is developed on the assumption that it is easy to recognize these minimum units.

2. Knowledge is said to be written by knowledge engineers. That is, knowledge about the environment that humans do not know is not given.

Further, in the system of Japanese Patent Laid-Open No. 6-274224, knowledge is acquired by learning, but since the structure of input / output data and neural network is general, it is not guaranteed that a hierarchical structure of knowledge can be obtained. Or, even if the neural network has the ability to acquire the hierarchical structure of knowledge, it is expected that a huge amount of time will be required.

Therefore, it is an object of the present invention to provide an image information processing method and apparatus capable of acquiring necessary image information at high speed.

It is another object of the present invention to provide various systems in which the image information processing method and its apparatus are effectively applied.

[0018]

According to the present invention, in a method of controlling an image information processing apparatus, an image is optically input from an image input unit of the image information processing apparatus, and feature points are detected from the input image. Then, the visual information amount is calculated based on the positions of the detected feature points, and the image input unit is controlled so that the visual information amount increases.

According to another aspect of the present invention, in the image information processing method, the input image is monitored, the evaluation value of each feature point in the input image is calculated, and the calculated evaluation value has a predetermined value. A feature point that exceeds the detected feature point is detected, the direction of the optical axis is moved toward the detected feature point, image data in the vicinity of the detected feature point is acquired, and an identifier is assigned to the acquired image data for detection. The positions of the selected feature points, the image data in the vicinity of the feature points, the detected time, and the assigned identifier are stored as a set.

According to another aspect of the present invention, the image information processing apparatus is provided with image input means for optically inputting an image, and detecting means for detecting feature points from the image input from the image input means. And a calculating means for calculating the amount of visual information based on the position of the feature point detected by the detecting means, and a control for controlling the image input means so that the amount of visual information calculated by the calculating means increases. And means.

According to another aspect of the present invention, the image information processing apparatus calculates a monitoring means for monitoring the input image, and an evaluation value of each feature point in the input image monitored by the monitoring means. Calculating means, a detecting means for detecting a feature point whose calculated evaluation value exceeds a predetermined value, a moving means for moving the direction of the optical axis toward the detected feature point, and the detecting means for detecting the feature point. Acquisition means for acquiring image data in the vicinity of the feature point, an identifier is allocated to the acquired image data, the position of the detected feature point, the image data in the vicinity of the feature point, and the time of detection are assigned. And a storage unit that stores the identifier as a set.

According to another aspect of the present invention, the image information processing apparatus is controlled by an input parameter, image input means for inputting an image, and the input image are discretized.
Mapping means for mapping into a multi-resolution space, feature point detecting means for detecting feature points from an input image, transform coding means for transforming the mapped image into a local pattern relating to the detected feature points, and transforming Quantizing means for quantizing the generated local pattern, knowledge acquiring means for obtaining a temporal and spatial correlation between the data quantized by the quantizing means, the quantized data and the correlation And input parameter control means for modifying the input parameter based on the above.

[0023]

DETAILED DESCRIPTION OF THE INVENTION Referring to the accompanying drawings,
A preferred embodiment according to the present invention will be described in detail.

[First Embodiment] FIG. 1 is a block diagram showing a configuration example of a visual information processing apparatus according to the present embodiment. Less than,
The operation of the components will be described in order.

<Lens system 1> Here, the lens system 1 is a wide view lens (fisheye lens).
(including fish eyelens), and a lens system for optically inputting an image, and a normal lens may be provided.

FIG. 2 shows an example of coordinate conversion by the wide-angle lens of the lens system 1. In the figure, x is the radius vector of the polar coordinate system on the image plane located in front of the input system, and t is the radius vector of the polar coordinate system after being converted by the wide-angle lens. Since this optical system stores the angle of this polar coordinate system (the angle θ of the polar coordinate system on the plane perpendicular to the paper surface), FIG. 2 and the following description will be limited to the radial direction.

X and t can be written as t = 2ξ tan (φ / 2) (1) x = ξ tan (φ) (2) using the incident angle φ with respect to the optical axis and the focal length ξ. From these, x can be expressed by the following equation as a function of t: x = t / {1- (t / 2ξ) ² } (3) This is the coordinate conversion in the wide-angle lens.

Radial frequency f spreading over the entire image plane
The pattern of (0) is applied to the array sensor 2 by the lens system 1.
When projected onto the plane of, the local radial frequency at the position of the distance t from the optical axis 10 on the plane of the array sensor 2 can be expressed by equation (4).

F (t) = f (0) · {(1- (t / 2ξ) ² ) ² / {1+ (t / 2ξ) ² } (4) In practice, the polar coordinate system (t, θ) is Using the two-dimensional image f
(T, θ) is output. Of course, when an arbitrary object existing in a three-dimensional real space is targeted, the incident angle φ
Since it suffices to know only the relationship between t and t, t = 2ξ tan (φ / 2) (5) is the coordinate transformation.

There are some wide-angle lenses that comply with coordinate conversion rules other than the above, and instead of the wide-angle lenses, these wide-angle lenses may be used in the lens system 1.

<Array sensor 2> The array sensor 2 samples the two-dimensional image f (t, θ) subjected to the coordinate conversion by the lens system 1 by the sensors arranged in a two-dimensional array, Obtain a discrete two-dimensional image. The sensor number at the upper left of the array sensor 2 is (1, 1), the m-th sensor number in the horizontal direction and the n-th sensor number in the vertical direction are described as (m, n). Output g _{m, n} of sensor with sensor number (m, n)
Is the integral kernel ψ _{m, n} (x),

[0032]

[Outside 1] Becomes As a result, {g _{m, n} } _{m, n} constitutes a discrete two-dimensional image.

<Two-dimensional filter (2D filter) 3> The two-dimensional filter 3 is an output signal of the array sensor 2, that is, a discrete 2D filter.
A three-dimensional image {g _{m, n} } _{m, n} is received as an input, the following mask processing is performed, and multi-resolution expansion is performed.

[0034]

[Outside 2] This mask has, for example, a ▽ ² G operator:

[0035]

[Outside 3] To use.

The following operator sets are also effective.

(A) A plurality of ∇ ² G operators having different spatial constants: A set of isotropic bandpass filters can be constructed.

(B) A plurality of Gabos having different spatial constants and directions
r operator: A set of bandpass filters that depend on direction can be constructed.

FIG. 3 shows a lens system 1, array sensors 2 and 2.
The relationship between the radial frequency band detected by a system in which the dimensional filter 3 is combined (hereinafter referred to as a wide-angle lens input device) and the distance from the optical axis 10 is schematically shown.
In the figure, the mask operator in the two-dimensional filter 3 is ▽ ²
In G, it is a constant case regardless of location. It is approximately equal to an image filter configured to remove the integral kernel that contributes to the detection of high frequency components as the distance from the optical axis increases. It can be seen from FIG. 3 that the entire area of the frequency space can be covered by changing the direction of the optical axis.

The above consideration is verified by the simulation results shown in FIG. From the figure, the center frequency detected by the wide-angle lens input device is, along with the displacement from the optical axis,
It can be confirmed that the signal decreases monotonically and that all components below the maximum frequency that the device can detect are detected.

<Feature Extractor 4>
The feature point detection unit 4 extracts local maximum points from the output image of the two-dimensional filter 3 as feature points and outputs their coordinates. The coordinates of these feature points are sent to the input parameter control unit 8 and used for determining the input parameter control amount.

Two-dimensional filter 3 and feature point detector 4
Can also be configured as follows by applying the principle of declination. In the two-dimensional filter, the vector {g _{m, n} } _{m, n} is first acted on by a gradient ▽ to generate a vector field {ω
Generate _{m, n} } _{m, n} : ω _{m, n} = ▽ g _{m, n} = {(g _{m, n} -g _{m-1, n} ) / Δx} e1 + {(g _{m, n} -g _{m , n-1} ) / Δy} e2 (9) where e1 and e2 are basis vectors, and Δx and Δy are horizontal and vertical grid intervals. Next, find the direction of the vector at each point (m, n): arg (ω _{m, n} ) ＝ arctan [{(g _{m, n} -g _{m, n-1} ) / △ y} / {(g _{m, n} −g _{m, n−1} ) / Δx}] (10) If the vector field is expressed by a complex function, the above direction is a complex argument. By applying the principle of declination to this, zeros and poles can be detected.

The principle of argument is that the function f (z) is a rational form in the singly connected region D, C is a positive closed simple closed curve in D, and C
It is assumed that there is no zero or pole of f (z) above.
Inside C, f (z) is a _j (j = 1, 2, ...,
m) and zeros of order λ _j , b _k (k = 1, 2,
, N) have poles of order μ _k , respectively. At this time, the increase of deflection angle of f (z) when returned to z ₀ to around the C starting from a point z ₀ on C △ _c arg f (z)
Is the number of zeros and poles in C (considering the order)
To

[0044]

[Outside 4] Can be given by the following formula: Δ _c arg f (z) = 2π (N _z (f) −N _p (f)) (11) The vector field {ω _{m, n} } A method for detecting the feature points of _{m, n} will be described below.

It is assumed that Step1 n = 1.

It is assumed that Step2 m = 1.

Consider an appropriate neighborhood around Step 3 (m, n). For example, 8 neighborhoods: {(m-1, n-1), (m, n-1), (m + 1, n-1), (m-1, n), (m + 1, n), ( m-1, n + 1), (m, n + 1), (m + 1, n + 1)} (12) is selected.

Step 5 By the principle of declination, the number of zeros and poles existing in the region surrounded by the above neighborhood is given as follows: N _z (ω) -N _p (ω) _{= (△ c arg ω m,} n) / 2π (13) △ c argω m, n is calculated as _{follows: △ c argω m, n ≒} Υ (ω m + 1, n + 1, ω m + _{1, n} ) + Υ (ω _{m, n + 1} , ω _{m + 1, n + 1} ) + Υ (ω _{m-1, n + 1} , ω _{m, n + 1} ) + Υ (ω _{m-1, n} , ω _{m-1, n + 1} ) + Υ (ω _{m-1, n-1} , ω _{m-1, n} ) + Υ (ω _{m, n-1} , ω _{m-1, n-1} ) + Υ (ω _{m + 1 , n-1} , ω _{m, n-1} ) + Υ (ω _{m + 1, n} , ω _{m + 1, n-1} ) (14) where Υ (x, y) = argx−argy if argx−argy ≤ π argy-argx otherwise (15)

Step 6 m = m + 1.

It is judged whether or not Step 7 m exceeds the range of the image. If it exceeds the range, the process proceeds to Step 8. If not, the process from Step 2 is repeated.

Step 8 n = n + 1.

It is judged whether or not Step 9 n exceeds the range of the image, and if it exceeds the range, the processing is terminated. If not, the processing from Step 3 is repeated.

The feature detection device detects a point at which the number N _z (ω) -N _p (ω) obtained from the above-mentioned principle of the angle of deviation becomes negative. As a result, a region having more poles than the zero point is detected, and a point having poles is detected in a sufficiently small region.

<Transform encoder 5>
The transform coding unit 5 has a function of transforming the image data, which is mapped on the multi-resolution space by the two-dimensional filter 3, into a local pattern space and outputting it.

First, the coordinates {s _f (= 2
^k ), b _f }, the set N _d (s _f , b _f ) of the neighborhoods up to the depth d
S = 2 ^p ; p = {k, k-1,…, Max (0, kd)} (16) b = b _f ± {(m-1 / 2) 2 ^p △ x, (n -1/2) 2 ^p △ y)}; m, n ＝ {1,…, 2 ^kp } (17) However, s _f and b _f are the scale (which may be considered as the reciprocal of the spatial frequency) and coordinates of the feature points detected by the feature point detection unit 4, and Δx and Δy are x of the array sensor 2.
The distance between the sensors in the direction and the y direction.

FIG. 5 shows N ₂ (s _f , b _f ) 51. _If the position (s _f , b _f ) in the multi-resolution space is different, N ₂ (s _f , b _f ) 51 is a region covered by the local pattern, that is, a real space region (b) as indicated by 52 and 53. It can be seen that the width) and the scale area (corresponding to the frequency area in the width of s) are different. That is, the area covered by N ₂ (s ₅₃ , b ₅₃ ) becomes the real space area 55 and the scale area 57,
The area covered by N ₂ (s ₅₂ , b ₅₂ ) is the real space area 54 and the scale area 56. Particularly, the vicinity of the depth "0" represents the pixel itself at the feature point position.

Thus, N _d (s _f , b _f ) is equal to a quadtree (a binary tree in the figure) whose root is the coordinate (s _f , b _f ) in the multiresolution space. Coordinates in multi-resolution space (s _f ,
local pattern P _d (s _f , b _f ) of depth d in b _f ).
Means the intensity corresponding to each node of N _d (s _f , b _f ), and the local pattern space of depth d defines an inner product for a finite dimensional vector in the set of quadtrees. It is a function space that is extended by things. By considering local patterns in multi-resolution space, invariants for certain motions of 3D objects are obtained. However, the depth (for example, how large the range is) depends on the object.

The data format output from the transform coding unit 5 is, for example, s = {s ₀ , s ₁ , s ₂ }, b = {b ₀ , b when only a local pattern of depth 2 is adopted. _For the discrete multiresolution space of ₁ …, b _j ,…, b _J }, {(b ₀ , (P ₂ (s ₀ , b ₀ ), P ₂ (s ₁ , b ₀ ), P ₂ ( s ₂ , b ₀ ))), (b ₁ , (P ₂ (s ₀ , b ₁ ), P ₂ (s ₁ , b ₁ ), P ₂ (s ₂ , b ₁ ))),… (b _J , (P ₂ (s ₀ , b _J ), P ₂ (s ₁ , b _J ), P ₂ (s ₂ , b _J )))} (18).

<Quantizer 6> FIG. 6 is a detailed block diagram of the quantizer 6. The quantizer 6 receives, for example, data in the following format from the transform encoder 5.

((B ₀ , (P ₂ (s ₀ , b ₀ ), P ₂ (s ₁ , b ₀ ), P ₂ (s ₂ , b ₀ ))), (b ₁ , (P ₂ (s ₀ , b ₁ ), P2 (s ₁ , b ₁ ), P ₂ (s ₂ , b ₁ ))),… (b _J , (P ₂ (s ₀ , b _J ), P ₂ (s ₁ , b _J ), P ₂ (s ₂ , b _J )))) (19) The quantizing unit 6 quantizes, for example, a local pattern of the above data and converts it into a code word S ₂ (s _f , b _f ) εZ. Yes: ((b ₀ , (S ₂ (s ₀ , b ₀ ), S ₂ (s ₁ , b ₀ ), S ₂ (s ₂ , b ₀ ))), (b ₁ , (S ₂ (s ₀ , b ₁ ), S ₂ (s ₁ , b ₁ ), S ₂ (s ₂ , b ₁ ))),… (b _J , (S ₂ (s ₀ , b _J ), S ₂ (s ₁ , b _J ), S ₂ (s ₂ , b _J )))} (20) Hereinafter, the processing procedure in the quantization unit 6 will be briefly described.

(A) Let j = 0.

(B) Data relating to the feature point bj in the quantizing unit 6, in the first case, (b ₀ , (P ₂ (s ₀ , b ₀ ), P ₂ (s ₁ , b ₀ ), P ₂ (s ₂ , b ₀ ))) (21) is input.

(C) P ₂ (s ₀ , b ₀ ), P ₂ (s ₁ , b ₀ ), P ₂ (s ₂ , b ₀ )
Is input to the quantizers 61, 62 and 63, and the corresponding codewords S ₂ (s ₀ , b ₀ ), S ₂ (s ₁ , b ₀ ), S ₂ (s ₂ , b ₀ ) are output. To be done. At this time, the codebook 64 is used for the quantizers 61, 62, and 63.

(D) (b ₀ , (S ₂ (s ₀ , b ₀ ), S ₂ (s ₁ , b ₀ ), S
₂ (s ₂ , b ₀ ))) is output.

(E) Return to (b) by setting j ← j + 1.

The quantizer 6 has a learning mode for obtaining a representative vector and an execution mode for encoding an input signal, which can be realized by a normal vector quantization technique.

Here, the code book 64 assigns numbers (code words) to the local patterns expressed as a set of component intensities at each node position.
It may be created by learning vector quantization as described in Example 2 below. Also, numbers may be sequentially assigned to all the local patterns that appear.

That is, the given image is encoded as a set of the position coordinates of the feature points and the code word of the local pattern. However, such coding involves considerable redundancy in the sense that there is a strong correlation between spatially adjacent local patterns. In the representative vector of the quantizer 6,
It is desirable not to include such redundancy. These redundancies can be reduced by using the simultaneous appearance probability between the respective representative vectors.

<Stochastic automato
n) 7> The set of the position coordinates of the feature points and the code word of the local pattern output from the quantizer 6 is input to each cell of the probability automaton 7.

FIG. 7 shows a configuration example of the probability automaton 7. In the figure, 71 is a pattern automaton constructed based on the geometrical characteristics and temporal correlation of image input data, and 72 is the result of the above pattern net and other input signals, for example, knowledge from a human keyboard. It is a symbolic automaton constructed from data and other sensor signals. 73 is a cell, which can take a finite number of states. A probability structure is defined on the set of state values. The stochastic automaton 7 has a hierarchical structure formally, but is generally said to be a block. The set of states of the r-th cell belonging to the q-th layer is Ω _r ^(q) , the probability distribution on it is {p (ω _u )}; ω _u ∈ Ω _r ^(q) , the v-th belonging to the (q + 1) -th layer of the set of states of the cell Ω _v ^{(q + 1),} the probability distribution on the {p (ω _z)}; written as ^{_{ω z ∈Ω v (q + 1}} ). At this time, it is assumed that these probability distributions are related by conditional probabilities as follows.

[0071]

[Outside 5] Here, p (ω _u) is the r-th cell belonging to the q layer ω _u
P (ω _z | ω _u ) is the probability that the v th cell belonging to the (q + 1) layer is ω _z when the r th cell belonging to the q layer is ω _u. (Conditional probability).

The cells belonging to the pattern automaton 71 are assigned one-to-one to each partial area when the multi-resolution space is divided. Therefore, the state value of the cell corresponds to the code word in the sub-region of the multi-resolution space. The state transition matrix having conditional probabilities as elements is calculated at the same time when the codebook 64 of the quantizer 6 is learned.

On the other hand, the cells belonging to the symbol automaton 72 have a one-to-one correspondence with objects or events. The conditional probabilities between them may be given by a knowledge engineer or calculated by the temporal and spatial correlation of the image input data.

For example, FIG. 8 shows an example of a three-level hierarchical representation in the case where multi-resolution representation is not performed by the transform coding unit 5. The coordination space at each level is composed of the (3 × 3) spatial arrangement of the coordination space at the next lower level. In other words, Ω ⁽⁰⁾ is the pixel that takes a real value (3 ×
The whole pattern formed by arranging in 3), Ω ⁽¹⁾ is an integer when the whole number attached to the pattern of Ω ⁽⁰⁾ is an integer,
The whole pattern made by arranging these integers in (3 × 3),
… And so on. Therefore, if there is a conditional probability between each layer as a model and the prior probability for 18 patterns belonging to Ω ⁽⁰⁾ is given, the probability distribution of the patterns belonging to Ω ⁽¹⁾ and Ω ⁽²⁾ can be calculated. .

The stochastic automaton 7 of the first embodiment and the mutual connection type neural network 207 of the second embodiment are collectively referred to as a knowledge acquisition unit.

<Input parameter control unit (input parameter
controller) 8> Based on the coordinates of the feature points from the feature point detection unit 4 and the probability distribution of the pattern from the probability automaton 7, input parameter control signals such as the optical axis direction of the lens and zooming to the wide-angle lens input device. It is an output component, and performs the following processing in the optical axis direction, for example.

The optical axis control method depends on which feature point is selected from the set of feature points detected by the feature point detecting section 4. For example, the evaluation criterion for this selection is defined by the following formula.

[0078] _{L br = L (w br,} T (b r, Ω v (q + 1)), ρ (b r)) (23) Here, w _br were normalized in the feature point b _r 2 The output value of the dimensional filter 3, the second term on the right side is the mutual information of the feature point b _r with respect to Ω _v ^{(q + 1)} , and ρ (b _r ) is b _r from the current optical axis.
Is the distance to A simplest example of L _br is a linear combination of the variables.

[0079] _{L br = α 0 w br +} α 1 T (b r, Ω v (q + 1)) of _{+ α 2 ρ (b r)} (24) First Ω _v Ω _r for the ^{(q + 1) (q)} Mutual information T (Ω _r ^(q) ,
Ω _v ^{(q + 1)} ) is defined by the following equation (25), and using this, T (b
Calculate _r , Ω _v ^{(q + 1)} ) = T (Ω _r ⁽⁰⁾ , Ω _v ^{(q + 1)} ).

[0080]

[Outside 6]

The correction amount of the optical axis is determined so that the optical axis coincides with the spatial coordinate that maximizes the above equation (23) or (24). Each cell of the pattern automaton 71 has both real space coordinates and scale coordinates. Therefore,
Matching the optical axis to the (observation) position with the maximum mutual information is the same as finding the cell with the maximum mutual information. Each cell stores the codeword and the occurrence probability for it, and since the relation between each cell is connected by the conditional probability, these are used to determine the state of the cell of a certain characteristic (this It is possible to determine the most effective cell, that is, the cell having the maximum mutual information, in order to lower the entropy with respect to the occurrence probability of the codeword of a specific cell. This is calculated using equations (23) through (25).

If the above equations (23) to (25) are applied to the multi-resolution space or its local pattern,
It also serves as an evaluation value for zoom control. The calculation and evaluation of the evaluation value of the present invention are not limited to Expression (23) to Expression (25).

<Specific Example of the Present Embodiment> The effectiveness of the present theory will be shown below by applying the present theory to a simple example. In this specific example, in order to simplify the explanation, (3 ×
An example will be described in which the array sensor of 3) is used and multi-resolution expression is not performed by the action of the wide-angle lens or the transform encoding unit.
In this example, the first term of the equation (24) is the change amount of pixel data (black /
It is considered to be white).

It is assumed that the input device can see a range of (3 × 3) pixels at a time and has a model as shown in FIG. 8 as knowledge. Here, an example of recognizing a two-dimensional pattern using only the pattern automaton will be shown. From the figure, it can be seen that the table of the level (0) pattern has 3 × 3 pixels, the table of the level (1) pattern has 9 × 9 pixels, and the table of the level (2) pattern has 27 × 27 pixels. Therefore,
The system can recognize the level (0) pattern with one observation. Also, the numbers appearing in the level (1) and level (2) patterns are the code words of the level (0) and level (1) patterns, respectively. Further, it is assumed that the appearance probabilities for the patterns of each level are equal. Hereinafter, the operation of recognizing the level (1) pattern will be referred to as level (1) observation.

Consider the problem of recognizing the level (2) pattern, given the pattern shown in FIG. 9 as the visual environment. However, in the figure, the upper left is the origin (0, 0) and the right is i
The coordinates of the j-th pixel below and below are denoted by (i, j). The system only knows two level (2) patterns, so the pattern in a given image need only identify this.

FIG. 9 shows an input image which is input to the present apparatus, and this image passes through the transform coding section 5 and the quantizing section 6 and becomes the format of the equation (13) (however, the redundant part is deleted). ).

In the cell at the top of the pattern automaton 71 of FIG. 7, there is a buffer for storing the appearance probabilities for the 18 codewords included in Ω ⁽⁰⁾ of FIG. Each cell in the first layer is a partial image (partial image of the input image) (3 × 3 partial image in this example) in the spatial area that it is responsible for, and 18 local areas included in Ω ⁽⁰⁾ in FIG. The pattern is matched and the corresponding codeword and the respective occurrence probabilities are stored.
When a partial image corresponding to a cell in the first layer is not obtained, the cell in the first layer assigns the same appearance probability “1/18” to all codewords. When it is found that the partial image has the local pattern “1” or “2”, the appearance probability of the code word 1 and the code word 2 is “1/2”.
, And “0” are stored as the appearance probabilities of other code words.

Each cell of the pattern automaton 71 stores code words and their appearance probabilities for the four patterns included in Ω ⁽¹⁾ in FIG. The pattern here is 1
This is the spatial arrangement of the codewords of the cells in the layer. That is, the local arrangement of the code words of the cells in the first layer is represented by the numbers "1" to "4". Which of the arrangements of the code words of the first layer from “1” to “4” in the subspace which the cell of the second layer has is calculated by using Expression (22). Formula (2
The multiplication of 2) is performed for 3 × 3 patterns in this example. Similarly, with respect to the cells of the third layer, the appearance probabilities for the two codewords are calculated using the equation (22).

Now, the initial level (0) observation is the coordinate (1
It is assumed that the processing is performed at the position of 0, 10), and the subsequent processing will be described step by step. The coordinates (10, 10) may be detected by the feature point detection unit.

(Step S1) The information obtained by the level (0) observation of the coordinates (10, 10) is the level (0).
It means that the pattern is "18", and it is easily understood that this information is "0". The system must first identify the level (1) pattern near the current observation point in order to associate the position currently observed by the system with the relative position of the level (2) pattern.

To end the observation of level (2), the entropy at Ω ⁽²⁾ must be minimized. That is, the observation point (i, j) that maximizes T (Ω _{(i, j)} ⁽¹⁾ , Ω _(10,10) ⁽²⁾ ) is selected. The observation point candidates are {(16,10), (13,13), (10,
16), (16, 16)} are selected. The expected value of the amount of information when observing each point is T (Ω _(16,10) ⁽¹⁾ , Ω _(10,10) ⁽²⁾ ) = _0.219 , T (Ω _(13,13) ⁽¹⁾ , Ω _(10,10) ⁽²⁾ ) = _0.354 , T (Ω _(10,16) ⁽¹⁾ , Ω _(10,10) ⁽²⁾ ) = 0.219, T (Ω _{( 16,16)} ⁽¹⁾ , Ω _(10,10) ⁽²⁾ ) = 0.354. (26) From this, if an observation point with a short distance is selected, (13, 13) is selected as the next observation point. The condition regarding the distance is not limited to this.

(Step S2) Level (0) observation is performed at coordinates (13, 13). As a result, the level (0) pattern “14” is obtained. This does not yet specify the level (1) pattern centered on the coordinate (13, 13). Therefore, similarly to step S1, level (0) observation at the coordinates (16, 16) is performed. As a result, the level (0) pattern "10" is obtained, and from this, the level (1) pattern centered on the coordinate (13, 13) is specified as "1". The amount of information at each observation point of level (2) is calculated again. Candidates for the observation points are the coordinates (2
2, 22). That is, it is specified whether the level (2) pattern at the coordinates (22, 22) is "2" or "3".
Therefore, the expected value of the amount of information when observing the level (0) candidate point is T (Ω _(19,19) ⁽¹⁾ , Ω _(16,16) ⁽²⁾ ) = 0. _{171, T (Ω (25,19)} (1), Ω (16,16) (2)) = 0. 171, T (Ω (22,22) (1), Ω (16,16) (2) ) = 0.585, T (Ω _(19,25) ⁽¹⁾ , Ω _(16,16) ⁽²⁾ ) = _0.171 , T (Ω _(25,25) ⁽¹⁾ , Ω _(16,16 ⁾ ₎ ⁽²⁾ ) = 0.585, from which the next observation point (22,22) is selected.

(Step S3) The level (0) pattern "17" is obtained by observing the coordinates (22, 22). From this, it can be seen that the level (1) pattern at this position is "3" or "4", and in the end, it is recognized that the given input signal is "2" of the level (2) pattern. it can.

In the above specific example, the multi-resolution space and its local pattern are not taken into consideration for simplification, but even if the multi-resolution space and its local pattern are taken into consideration, the processing becomes complicated. Similar processing is possible. By taking into consideration the multi-resolution space and its local pattern, zoom control and the like can be performed in addition to the control of the optical axis.

[Second Embodiment] In the second embodiment, an example in which the stochastic automaton 7 of the first embodiment is realized by a mutual connection type neural network (Hopfield-type NN) will be described. As a result, the processing in the input parameter control unit 208 also changes.

FIG. 10 is a block diagram showing an example of the configuration of the visual information processing apparatus according to this embodiment. The lens system 201, array sensor 202, two-dimensional filter 203, feature point detection unit 204, and transform coding unit 205 are , The lens system 1, the array sensor 2, the two-dimensional filter 3 of the first embodiment, respectively.
This is the same as the feature point detection unit 4 and the transform coding unit 5.

<Quantization Unit 206> The quantization unit 20 shown in FIG.
6 shows the configuration of No. 6. The quantizer 206 has a learning mode for acquiring a representative vector and an execution mode for encoding an input signal. There are two methods for obtaining the representative vector in the learning mode: a method using a correlation matrix and a method proposed by Kohonen using a learning vector quantizer. In the following, a method of constructing a quantizer of depth m by the Kohonen learning vector quantizer will be described.

(A) Prepare weight vectors W _n ^m for the number of representative vectors, and initialize each with a small random number.

(B) The same number of arithmetic elements (pr) as the weight vector W _n ^m , which outputs the inner product of the local patterns X ^m and W _n ^m.
ocessing element) PE _n ^m is prepared. 2 between each PE
The dimension distance is introduced, and the neighborhood radius R of PE is set appropriately.

(C) For the input local pattern X ^m , PE _n ^m _MAX that outputs the maximum value is found, and PE _n ^m is found.
The distance between the _MAX modifies the weight vector W _n ^m for R less than PE by the following equation.

[0101] _{^{W n m ← W n m +}} η W (X m -W n m MAX) here η _W is a constant. The radius R near the PE is gradually reduced as the weight vector is corrected.

In this way, the local pattern having a high appearance probability is stored in the learning vector quantizer as a weight vector. The coding of the local pattern is given as the index of the PE that outputs the maximum value.

In the execution mode of the quantizing unit 206, the feature vector in the vicinity of each feature point is input to the above-mentioned learning vector quantizer, and among the {PE _n ^m } _n at that time, the arithmetic element with the maximum output The index of is the codeword. By this processing, all the feature points are associated with appropriate code word sets.

In this way, the given image is coded as a set of the position coordinates of the feature points and the code word of the local pattern. However, such coding involves considerable redundancy in the sense that there is a strong correlation between spatially adjacent local patterns. It is desirable that the representative vector of the quantizer 206 does not include such redundancy. These redundancies can be reduced by using the simultaneous appearance probability between the respective representative vectors.

<Mutually Connected Neural Net (Hopfield
-type neural network) 207> FIG. 12 shows the configuration of the mutual connection type neural network 207. In FIG. 12, a pattern net 121 is a network constructed on the basis of geometrical characteristics and temporal correlation of image input data,
The concept net 122 is a network constructed by the result of the pattern net and other input signals, for example, knowledge data from a keyboard by a human or other sensor signals.

The neuron 123 is a multi-input / single-output arithmetic element, and the input vector x and the output value y are connected using the weight vector w in the relation of y = sgm (w ^T x) (28). . However, sgm (•) is a sigmoid function. The output value of the neuron represents the appearance probability of the pattern or concept corresponding to the neuron.

Reference numeral 124 is a neuron group as a set of neurons competing with each other, and 125 is a weight W _{j, k} connecting the neuron j and the neuron k. The neuron group formally has a hierarchical structure, and each hierarchy forms a plurality of blocks. The neuron group of the (q) layer r block is written as Ω ^{(q): r} , and the neuron group of the (q + 1) layer v block is written as Ω ^{(q + 1): v} . The group of neurons belonging to the pattern net 121 expresses a local pattern space at corresponding coordinates in the multi-resolution space. Data from the quantization unit 206 is set in the pattern net 121 as an initial state.

The motion of the neural network 207 having the above structure is governed by the energy function defined by the following equation. That is, the output value of the neuron belonging to each block is V _m εΩ
If we write ^{(q): r} , V _n ∈Ω ^{(q + 1): v,} we can define as follows:

[0109]

[Outside 7]

Since the weight of the pattern net 121 corresponds to the weight in the learning vector quantizer, the weight value obtained by the learning vector quantization can be used, but it can also be obtained by the following method.

The pattern net 121 and the concept net 12
The 2 weights can be obtained as follows. That is, by considering equation (29) as a learning potential function, the weight correction amount is

[0112]

[Outside 8] Becomes

<Input Parameter Control Unit 208> The mutual information amount described in the first embodiment is calculated as follows in correspondence with the mutual connection type neural network.

(A) A set of neurons in the (q) layer r block of the mutual connection type neural network V _m ∈Ω ^{(q): r}
And the set of neurons in the (q + 1) -layer v block V _n ∈Ω
^The entropy ε _{(q): r} ^{(q + 1): v} of the set W _m ⁿ of connections connecting ^{(q + 1): v} with

[0115]

[Outside 9] Calculate by here,

[0116]

[Outside 10] Is a value calculated in the learning mode, and may be treated as a known value in the input parameter calculation process.

(B) Mutually connected neural network
(Q) Layer r block neuron set V _m ∈Ω ^{(q): r}
For entropy ε _{(q): r} ,

[0118]

[Outside 10] Calculate by

(C) From ε _{(q): r} ^{(q + 1): v} and ε _{(q): r} , the same value as in the equation (25) is obtained as follows.

T (ε ^{(q), T + 1} , p (ω; A _T ^{(q + 1)} )) = − ε _{(q): r} ^{(q + 1): v} + ε _{(q): r} (33 )

[Third Embodiment] In the third embodiment, an example of application of the visual information processing device of the present embodiment to a monitoring device will be described.

FIG. 13 is a block diagram showing a configuration example of the monitoring apparatus of this embodiment. The lens system 301, the array sensor 302, and the two-dimensional filter 303 are the lens system 1, the array sensor 2, and the array sensor 2 of the first embodiment. It is similar to the dimensional filter 3.

<Characteristic point detection unit 304> Characteristic point detection unit 30
In No. 4, not only the feature points related to the spatial arrangement by the two-dimensional filter 303 but also the extreme value of the temporal change of the image data are detected as the feature points. The monitoring device is installed in a place where there is almost nothing that normally moves, such as in a store or office. If there is no change, the image data will not change, so
Only the changing places need to be observed in detail. This can significantly lengthen the recording time of the monitoring information.

From such a meaning, the temporal change of image data is an important feature. In order to capture a moving object, we want to minimize the time delay of feedback control. The feature point detection unit 304 also has a function of calculating the optical axis control amount for directing the optical axis to the detected feature point position and sending it to the input parameter control unit 308. With such a function, the image of the moving object on the sensor can be processed as a still image.

<Transform Encoding Unit 305> Encodes image data relating to the captured moving object. This encoded data is stored in the external storage device 309 via a communication line, a bus line, or the like.

<Quantizer 306> The coded data of the captured object is immediately assigned to an appropriate codeword. Therefore, when these objects subsequently appear on the image, they are represented by a position and a corresponding codeword.

<Probability Automata 307> The coded data of the captured object is immediately reflected in the state transition probability matrix of the probability automaton 307. By knowing the strength of a limited frequency domain, an object containing that frequency strength function can be estimated. As a result, the position of the object once captured does not need high-resolution data with the optical axis aligned, and may be observed with any part of the wide-angle lens.
That is, there is no need to control the optical axis. Even if a plurality of objects appear in the monitoring area, they may be observed only once in order.

<Input Parameter Control Unit 308> As an evaluation criterion for selecting a feature point, a linear combination of a two-dimensional filter output value at a feature point, a time change amount, and a mutual information amount is adopted.

With respect to a moving object, if the standard is set such that it is observed at the center of the optical axis at the time when it appears and that the object is not captured at the center of the optical axis thereafter,
A more efficient monitoring device is possible.

<External Storage 309> Stores various data. The monitoring device first stores the image data in a normal state. After that, only the part that changes with time is stored.
The stored data is the coded data of the captured object, the captured time, and the temporal changes (trajectory) of their positions.
With such a storage form, the amount of data to be stored can be greatly reduced, and thus recording for a long time becomes possible.

<Operation Example of Monitoring Device> The actual operation of the monitoring device configured as described above will be described with reference to the flowchart of FIG.

First, in step S11, the initial image of the installed place is stored in the storage unit 309.

Next, in step S12, the characteristic points are evaluated by the equation (24). As is clear from the equation (24), the feature point having a large evaluation value is whether the temporal change amount of the pixel values in the vicinity thereof is large or the obtained information is large. Even when there are a plurality of feature points having the same evaluation value of the above two items, the equation (24) is defined so that the one closest to the optical axis is selected from them.

In step S13, if there is a feature point whose evaluation value is larger than a certain threshold value α,
Go to step S16.

If it is determined in step S14 that the amount of change over time of all feature points is smaller than a certain threshold β, step S14.
Return to 12.

In step S15, among the feature points, those whose amount of change over time exceeds a certain threshold value β are related to an object described later based on the codeword vector (which may be incomplete) corresponding to the feature point. The object existing there is identified using the complete description, the current time, the coordinates of the feature point, and the code word corresponding to the object are paired and stored in the storage unit 309, and the process returns to step S12.

In step S16, the optical axis is made to coincide with the selected feature point.

In step S17, a complete description (codeword set) of the object existing in the vicinity of the feature point is obtained, and an appropriate number is assigned. The storage unit 30 combines the number assigned to the object, the set of code words, the time, and the feature point coordinates as a set.
Store in 9. It returns to step S12.

The shift to step S15 corresponds to the case where, based on the measurement data in the vicinity of the feature point, it can be estimated with high reliability that the data was previously observed and stored as detailed data. Therefore, the monitoring device of the present embodiment performs detailed observation only when someone enters the store (S16, S17) and thereafter stores only the position information of that person (S15).

By using such a monitoring device, it is possible to monitor for a long time with a storage device having a capacity much smaller than that of storing a moving image as it is. For example, one image is 2
In the case of 60 Kbytes (512 × 512 pixels, 1 pixel is 1 byte), 28 Gbytes are required to store 108,000 image columns (30 images / sec for 1 hour). According to the present embodiment, for example, assuming that there are always 100 moving objects (customers in a store), even if one screen (260 Kbytes) is required to completely describe each object, one hour of storage is required. The required capacity is 0. 26 MB, 26 MB for the description of the object, 1. The sum is 728 Mbytes (however, the position of the feature point is described in double-precision two-dimensional coordinates), that is, 28 Mbytes. In reality, it is unlikely that there will be 100 people in the store all the time, and the capacity used to describe an object is smaller than the capacity that stores the entire image. For example, even if the number of objects in the store is reduced to 10, the capacity required for storing one hour is drastically reduced to 3 Mbytes.

[Fourth Embodiment] In the fourth embodiment, the quantizing unit 6 of the first embodiment is realized by a learning local vector quantizer, the probability automaton 7 is realized by a symmetric connection type neural network group, and a signal from the outside is realized. An example in which a signal input unit for inputting a signal and a signal output unit for outputting a signal to the outside are provided is shown. This also changes the processing in the input parameter control unit.

FIG. 15 is a block diagram showing an example of the arrangement of the visual information processing apparatus according to this embodiment, which is an image input section 401.
Includes a lens system and an array sensor similar to the lens system 1 and the array sensor 2 of the first embodiment. Two-dimensional filter 403, feature point detection unit 404, transform coding unit 405
Are the same as the two-dimensional filter 3, the feature point detection unit 4, and the transform coding unit 5 of the first embodiment, respectively.

<Quantization Unit 406> FIG. 16 shows the quantization unit 4
The configuration of the learning local vector quantizer (LLVQ) that becomes 06 is shown. LLVQ has a learning mode for obtaining a representative vector and an execution mode for encoding an input signal. To acquire the representative vector in learning mode,
There are a method using a correlation matrix and a method using a learning vector quantizer proposed by Kohonen. In the following, with the learning vector quantizer of Kohonen, the depth m
The method of constructing the quantizer of is described.

(A) The weight storage units 162 are prepared in a predetermined number and store representative vectors W _n belonging to a certain category n. FIG. 16 shows a case where there are two representative vectors.

(B) The weight updating unit 163 is provided for each weight storage unit 162, and the weight storage unit 162 is provided only when the signal transmitted from the binarization unit 164 is 1.
The weight value stored in is updated according to the following equation (34).

[0146] _{^{W n m ← W n m +}} η W (W n m -X m) (34)

(C) The inner product calculation unit 161 calculates the inner product of the input signals X and W _n and sends it to the binarization unit 164.

(D) The binarization unit 164 receives the output signals of the plurality of inner product calculation units 161, and converts the output value having the maximum value into 1 and the others into 0. The converted signals (binary signals) are transmitted to the corresponding weight storage update units 163. In addition, a set of these binary signals is transmitted to the symmetrical connection type neural network group 406.

In this way, the given image is encoded as a set of the position coordinates of the feature points and the code word of the local pattern. However, such coding involves considerable redundancy in the sense that there is a strong correlation between spatially adjacent local patterns. It is desirable that the representative vector of the quantizer 405 does not include such redundancy. These redundancies can be reduced by using the simultaneous appearance probability between the respective representative vectors.

<Symmetrically Connected Neural Net Group 406>
FIG. 17 shows the configuration of the symmetrical connection type neural network group 406. The symmetric connection type neural net group 406 is configured by a plurality of symmetric connection type neural nets that transmit signals to each other, and each symmetric connection type neural network is another symmetric connection type neural network, the quantizing unit 405, or a signal input. The signal is received from the unit 408, and the processing result of the input signal is output to another symmetrical connection type neural network, the input parameter control unit 407, or the signal output unit 409.

FIG. 18 shows the structure of one symmetric connection type neural network. In FIG. 18, in the target connection type neural network state updating unit 171, neurons having multiple inputs and one output and having a sigmoid function as a nonlinear input / output function are connected to each other via weights. A neuron forms at least two blocks, one of which is an output block and the rest are all input blocks. The set of output values of the neurons of the output block becomes an output signal (vector), and the neuron belonging to the input block receives a signal from another symmetric connection type neural network, the quantization unit 405, or the signal input unit 408.

The motion of the neural network 207 having the above structure is governed by the energy function defined by the following equation. That is, the output value of the neuron belonging to each output block is V
_n, an output value V _m of neurons belonging to any one of the input block, an input value I _m, to write the weight between these neurons W _m, and _n, the following equation energy function H to these (35) Can be defined as:

[0153]

[Outside 12]

The weight updating unit 172 uses, for example, the following equation (3
The weight of the neural network is updated based on the weight update rule of 0).

[0155]

[Outside 13] The weight storage unit 173 stores the weight updated by the weight update unit 172.

The first information amount calculation unit 174 calculates the first information amount ε based on the output value V _{n of the} neuron belonging to the output block.
Calculate ₁ as follows:

[0157]

[Outside 14] The second information amount calculation unit 175 sets the output value V _{n of} the neuron belonging to the output block, the output value V _{m of the} neuron belonging to the input block, and the weight between these neurons to W.
The second information amount ε ₂ based on _{m, n} is calculated as follows.

[0158]

[Outside 15] here,

[0159]

[Outside 16] Since is a value calculated in the learning mode, it may be treated as a known value in the input parameter calculation process.

The third information amount calculation unit 176 calculates the first information amount ε.
The third information amount T is calculated from ₁ and the second information amount ε ₂ as follows.

T = −ε ₂ + ε ₁ (39)

<Input Parameter Control Unit 407> The input parameter control unit 407 instructs the image input unit 401 based on the coordinates of the feature points from the feature point detection unit 403 and the state value of the symmetric connection type neural network group 406. , Outputs an input parameter control signal for the optical axis direction of the lens, zooming, and the like.

For example, the control of the optical axis direction is determined by which feature point is selected from the set of feature points detected by the feature point detecting section 403. This selection criterion is, for example, the following formula (40)
Defined by

L _j = L (ω _j , T _j , ρ _j ) (40) where ω _j is the output value of the normalized two-dimensional filter near the j-th feature point, and T _j is the j-th The third amount of information near the feature point, ρ _j, is the distance from the current optical axis to the j-th feature point.

Further, the signal input unit 408 inputs an input signal from another device or the like, for example, knowledge data by a human from a keyboard or another sensor signal. Signal output unit 4
Reference numeral 09 designates an output signal of the symmetric connection type neural network group 406 as an input parameter control unit 407 of the visual information processing device.
Other unit other than, or output to other devices.

As described above, by combining with another device, the visual information processing device of the present embodiment is
Various characteristic uses are possible.

For example, in combination with a wireless signal device or a wired signal device, when observing a human face, if the person's name is input by a wired signal or a wireless signal,
Face image data and a name can be stored in association with each other. This is not limited to human faces in the sense of tagging images. Furthermore, by associating with a voice signal in combination with a voice signal input device, the recognition accuracy of an image is improved.

The audio signal input device and the audio signal processing device are
In particular, it is not limited to human voice, and ultrasonic waves may be used. In this case, since the ultrasonic sensor can obtain the distance information to the surrounding object, the autonomous traveling robot can be configured using the visual information and the distance information.

The wireless signal device and the wireless signal processing device can be used for communication between a plurality of autonomous traveling robots or a plurality of monitoring devices. For example, consider a case where the inside of a building cannot be monitored by a single monitoring device. It is assumed that the monitoring device j detects a moving object and makes a detailed observation thereof. When this moving object leaves the observable region of the monitoring device j and is about to enter the observable region of the monitoring device k, the monitoring device j tells the monitoring device k, “Now, the monitoring device j.
Detailed data of a moving object that has entered the observable region of the monitoring device k from the observing region of the monitoring device k is acquired by the monitoring device j.
Then, it is not necessary to perform detailed observation of this moving object. This is effective in reducing the amount of stored data and the work of the monitoring device.

Furthermore, when combined with a traveling robot for transporting parts and the like in a factory, for example, a traveling robot flexibly adapting to the environment can be realized. It can also be combined with an audio signal output device.

According to the embodiments described above, it is possible to provide an image information processing method and an apparatus therefor capable of acquiring optimal visual information at high speed, and a control method therefor, and it is also possible to provide various apparatuses to which the apparatus is effectively applied. .

More specifically, 1. Even if the subsequent image input becomes difficult for some reason, the result with the highest appearance probability can be obtained. Furthermore, since these can be performed in parallel, the processing time can be greatly reduced. Furthermore, since it is realized by the mutual connection type neural network, the state transition rule is simplified and the processing time can be expected to be further improved.

2. The feature point set of the image is preferably a finite set of points. However, edges that are widely used as feature quantities in the conventional technique are continuous, and in order to obtain a finite number of point sets based on this, some kind of post-processing, such as binarization processing for edge strength, is performed. Is required. According to the present invention, a singular point can be obtained as an isolated point. Further, the transform coding unit, the quantization unit, and the knowledge acquisition unit can handle the sensor input and the knowledge data in a unified manner, and thus can be applied to the recognition of an environment composed of complicated visual patterns. You can

Laplaci as a two-dimensional filter
By using the an-Gussian filter, the visual information of the environment can be approximately converted on the partial region of the multi-resolution space. By appropriately controlling the direction of the optical axis, a description in a multi-resolution space for an image or a partial region of the environment can be approximately obtained. If a simple averaging process is used as the two-dimensional filter, it is possible to obtain a description in a multi-resolution approximation for a partial region of an image or environment. In this way, by using the one used in the multi-resolution space as the core of the two-dimensional filter, a multi-resolution space corresponding to it can be obtained. This is an effective sensing method when the visual information of the environment covers a partial area of the multi-resolution space.

4. The feedforward type neural network enables faster feedforward control than the mutual connection type neural network.

5. By adopting a multi-resolution space as a feature space and focusing on the local pattern there,
It is possible to obtain an invariant for a specific movement of an object existing in the environment. For example, movement along the optical axis only translates the feature quantity of the object along the scale axis.

6. By combining a voice signal input device and a voice signal processing device, it becomes possible to send commands from humans by voice, or to identify obstacles by sound, etc., and to use a wireless signal communication device and a wireless signal processing device, or a wired signal. By combining the communication device and the wired signal processing device, it becomes possible to send and receive commands with other computers, or send and receive information with other visual information processing devices,
The knowledge acquisition unit can perform input parameter control based on the voice signal, the wireless signal, and the wired signal,
Objects that could not be recognized with only a single signal can be recognized. Further, by receiving the knowledge data of another visual information processing device, the time required for the learning mode can be shortened.

7. By combining a work tool and the work tool control device, it is possible to perform a work that is adapted to the external environment, and an autonomous work that effectively operates when the way of performing the work largely depends on the environment. Allows for collaborative work by robots.

8. By combining with an audio signal output device, it is possible to configure an autonomous work robot that can inform surrounding people of its own state and can also work in collaboration with humans.

The present invention described above may be applied to a system composed of a plurality of devices or to a specific device within the system. Further, the present invention can be applied when this device is achieved by executing a program, and the program may be supplied from an external storage medium, and the storage medium storing the program is also within the scope of the present invention. It is in.

[0181]

As described above, according to the present invention,
It is possible to provide an image information processing method and apparatus that can obtain optimal visual information at high speed, and a control method thereof.

[Brief description of drawings]

FIG. 1 is a diagram showing a block configuration of a visual information processing device according to a first embodiment.

FIG. 2 is a diagram for explaining coordinate conversion by a wide-angle lens.

FIG. 3 is a diagram for explaining frequency characteristics of a wide-angle lens.

FIG. 4 is a diagram for explaining a spatial frequency detected by a sensor.

FIG. 5 is a diagram for explaining an example of a local pattern on a multi-resolution space.

FIG. 6 is a diagram illustrating a configuration example of a quantization unit used in the first embodiment.

FIG. 7 is a diagram illustrating an example of a probability automaton according to the first embodiment.

FIG. 8 is a diagram showing an example of a hierarchical representation used in a specific example.

FIG. 9 is a diagram showing an example of a visual environment used in a specific example.

FIG. 10 is a block diagram showing a configuration example of a visual information processing device according to a second embodiment.

FIG. 11 is a diagram illustrating a configuration example of a quantization unit according to the second embodiment.

FIG. 12 is a diagram illustrating an example of a mutual connection type neural network.

FIG. 13 is a block diagram illustrating a configuration example of a monitoring device according to a third embodiment.

FIG. 14 is a flowchart illustrating an operation example of the monitoring device according to the third exemplary embodiment.

FIG. 15 is a block diagram showing a configuration example of a visual information processing device according to a fourth embodiment.

FIG. 16 is a diagram illustrating a configuration example of a quantization unit according to the fourth embodiment.

FIG. 17 is a diagram showing a configuration example of a symmetrical connection type neural network group.

FIG. 18 is a diagram showing a configuration example of a symmetrical connection type neural network.

[Explanation of symbols]

 1, 201, 301 Lens system 2, 202, 302 Array sensor 3, 203, 303, 402 Two-dimensional filter 4, 204, 304, 403 Feature point detection unit 5, 205, 305, 404 Transform coding unit 6, 206, 306 Quantizer 7, 307 Probability automaton 8, 208, 308, 407 Input parameter controller 10 Optical axis 61, 62, 63 Quantizer 64 Codebook 71 Pattern automaton 72 Symbol automaton 73 Cell 121 Pattern network 122 Concept net 123 Neuron 124 neuron group 125 weight 161 inner product calculation unit 162, 173 weight storage unit 163, 172 weight update unit 164 binarization unit 171 symmetric connection type neural network update unit 174 first information amount calculation unit 175 second information amount calculation unit 176 Third information amount calculation unit 207 Mutual connection type neural network 309 Storage unit 401 Image input unit 406 Symmetric connection type neural network group 408 Signal input unit

Claims (50)

[Claims] 1. An image is optically input from an image input unit of an image information processing apparatus, a feature point is detected from the input image, and a visual information amount is calculated based on the position of the detected feature point. A method for controlling an image information processing apparatus, comprising controlling the image input unit so that the amount of visual information increases. 2. The image information in the vicinity of the feature point is extracted, and the visual information amount is calculated based on the position of the feature point and the extracted image information in the vicinity of the feature point. Item 2. The control method according to Item 1. 3. The control method according to claim 1, wherein an optical axis of the image input unit is moved to control the image input unit. 4. The control according to claim 3, wherein the visual information amount is calculated based on the position of the feature point, image information in the vicinity of the feature point, and the distance between the feature point and the optical axis. Method. 5. The control method according to claim 1, wherein zooming of the image input unit is performed to control the image input unit. 6. The control method according to claim 1, wherein the visual information amount is calculated based on a mutual information amount between different cells in a probability automan including a plurality of cells. 7. The control method according to claim 1, wherein the visual information amount is calculated based on entropy in the mutual connection type neural network. 8. The control method according to claim 2, wherein image information in the vicinity of the feature point in the multi-resolution space up to a predetermined depth is extracted as the image information in the vicinity of the feature point. 9. The step of extracting image information in the vicinity of the predetermined depth includes a step of discretizing an input image and a step of masking the discretized two-dimensional image. Item 9. The control method according to Item 8. 10. The control method according to claim 1, wherein the step of inputting the image includes the step of transforming the coordinate of the input image in a direction perpendicular to the wide-angle lens using a wide-angle lens. 11. The step of detecting the feature points includes the step of generating a vector field from a function representing an input image, the vector field being expressed by a complex function, and using the argument principle as a singular point. The control method according to claim 1, further comprising the step of detecting a point. 12. The control method according to claim 1, wherein the visual information amount is calculated by a plurality of symmetrically coupled neural networks that transmit and receive signals to and from each other. 13. The symmetric connection type neural network includes one output neuron block and at least one input neuron block, and the step of calculating the visual information amount is performed by using an output value of the output neuron block. Calculating a first amount of information based on the output value of the input neuron block, and calculating a second amount of information based on the weight of the connection between the input neuron block and the output neuron block. And a step of calculating the visual information amount based on the first information amount and the second information amount.
2. The control method described in 2. 14. A feature in which an input image is monitored, an evaluation value of each feature point in the input image is calculated, a feature point whose calculated evaluation value exceeds a predetermined value is detected, and a direction of an optical axis is detected. Moving toward a point, acquiring image data in the vicinity of the detected feature point, assigning an identifier to the acquired image data, detecting the position of the detected feature point, and image data in the vicinity of the feature point; An image information processing method characterized by storing a detected time and an assigned identifier as a set. 15. A feature point having a calculated evaluation value exceeding a second predetermined value is detected, and an identifier assigned to image data in the vicinity of the feature point is identified from the position of the detected feature point, The image information processing method according to claim 14, wherein the position of the detected feature point, the detected time, and the identified identifier are stored as a set. 16. The image information processing method according to claim 14, wherein the evaluation value is calculated based on the position of the feature point and the image data in the vicinity of the feature point. 17. When there are a plurality of feature points detected in the step of detecting the feature points, in the step of moving in the direction of the optical axis, the light is moved toward the feature point closest to the current optical axis. The image information processing method according to claim 14, wherein the axis is moved. 18. In the step of calculating the evaluation value, the evaluation value is calculated based on the position of the feature point, image information in the vicinity of the feature point, and the distance between the position of the feature point and the optical axis. 15. The image information processing method according to claim 14. 19. An image inputting means for optically inputting an image, a detecting means for detecting a characteristic point from the image input from the image inputting means, and a position of the characteristic point detected by the detecting means. An image information processing apparatus comprising: a calculation unit that calculates the amount of visual information; and a control unit that controls the image input unit so that the amount of visual information calculated by the calculation unit increases. 20. An extracting means for extracting image information in the vicinity of the feature point, wherein the calculating means calculates the visual information amount based on the position of the feature point and the image information in the vicinity of the feature point. The image information processing apparatus according to claim 19, wherein the image information processing apparatus is provided. 21. The image information processing apparatus according to claim 19, wherein the control unit moves the optical axis of the image input unit. 22. The calculation means calculates the amount of visual information based on the position of the feature point, image information in the vicinity of the feature point, and the distance between the feature point and the optical axis. The image information processing apparatus according to claim 21. 23. The image information processing apparatus according to claim 19, wherein the control unit zooms the image input unit. 24. The image according to claim 19, wherein the calculating means includes a probability automan including a plurality of cells, and calculates the visual information amount based on mutual information amount between different cells. Information processing equipment. 25. The image information processing apparatus according to claim 19, wherein the calculation means includes an interconnected neural network, and calculates the visual information amount based on the entropy of the network. 26. The image information processing according to claim 19, wherein the extraction means extracts image data in the vicinity of the feature point in the multi-resolution space up to a predetermined depth as the image data in the vicinity of the feature point. apparatus. 27. The extracting means includes discretizing means for discretizing an input image, and a filter for masking the discretized two-dimensional image.
The image information processing apparatus described in 1. 28. The input means includes a wide-angle lens,
20. The image information processing apparatus according to claim 19, wherein the input image is subjected to coordinate conversion in a direction perpendicular to the wide-angle lens. 29. The detecting means detects a characteristic point as a singular point by using a generating means for generating a vector field from a function representing an input image, expressing the vector field by a complex function, and using the principle of argument. 20. The image information processing apparatus according to claim 19, further comprising: 30. The image information processing apparatus according to claim 19, wherein the calculation unit includes a plurality of symmetrically coupled neural networks that transmit and receive signals to and from each other. 31. The symmetrical connection type neural network includes one output neuron block and at least one input neuron block, and the calculating means outputs a first output neuron block based on an output value of the output neuron block. First calculation means for calculating the information amount, second calculation means for calculating the second information amount based on the output value of the input neuron block and the weight of the connection between the input neuron block and the output neuron block 32. The image information processing apparatus according to claim 31, further comprising: a third calculation unit that calculates the visual information amount based on the first information amount and the second information amount. 32. A monitoring means for monitoring the input image, a calculating means for calculating an evaluation value of each feature point in the input image monitored by the monitoring means, and a characteristic for which the calculated evaluation value exceeds a predetermined value. Detecting means for detecting a point, moving means for moving the direction of the optical axis toward the detected characteristic point, acquiring means for acquiring image data in the vicinity of the characteristic point detected by the detecting means, An image characterized by a storage means for assigning an identifier to the image data, and storing the position of the detected feature point, image data in the vicinity of the feature point, the time of detection, and the assigned identifier as a set. Information processing equipment. 33. The second detecting means for detecting a feature point whose evaluation value calculated by the calculating means exceeds a second predetermined value, and the feature from the position of the feature point detected by the second detecting means. The identification means for identifying the identifier assigned to the image data near the point, the second storage means for storing the position of the detected feature point, the detection time, and the identified identifier as a set, The image information processing apparatus according to claim 32, 34. The image information processing apparatus according to claim 32, wherein the calculating means calculates the evaluation value based on a position of a feature point and image data in the vicinity of the feature point. 35. When there are a plurality of characteristic points detected by the detecting means, the moving means moves the optical axis toward the characteristic point closest to the current optical axis. Item 32. The image information processing device. 36. The calculating means calculates the evaluation value based on a position of a characteristic point, image data in the vicinity of the characteristic point, and a distance between the position of the characteristic point and the optical axis. The image information processing apparatus according to claim 32. 37. Image input means controlled by an input parameter, for inputting an image, mapping means for discretizing the input image and mapping it in a multi-resolution space, and feature inspection for detecting feature points from the input image. Output means, transform coding means for transforming the mapped image into a local pattern relating to the detected feature points, quantizing means for quantizing the transformed local pattern, and quantizing means by the quantizing means. An image information processing apparatus comprising: knowledge acquisition means for obtaining a temporal / spatial correlation between these data, and input parameter control means for correcting the input parameter based on the quantized data and the correlation. . 38. The image information processing apparatus according to claim 37, wherein the knowledge acquisition means further uses knowledge data from the outside. 39. The knowledge acquisition means comprises a probability automaton having a cell to which the data quantized by the quantization means is input, and stores the correlation as a state value of a cell of the probability automaton. The image information processing apparatus according to claim 37, wherein 40. The input parameter control means calculates the correction amount of the input parameter based on the image data of the feature point, the state value of the cell relating to the feature point, and the distance between the feature point and the optical axis. 40. The image information processing device according to claim 39, wherein: 41. The image information processing apparatus according to claim 39, wherein the probabilistic automaton is configured by using an interconnected neural network. 42. The feature point detecting means includes a vector field generating means for generating a vector field of an input signal, and a singular point detecting means for detecting a singular point from the vector field using a principle of declination. 40. The image information processing device according to claim 39. 43. The image input means comprises a wide-angle lens or a fish-eye lens, an array sensor for converting an image passing through the lens into a discrete two-dimensional image, and a two-dimensional filter for masking the discrete two-dimensional image. 37. The method according to claim 37,
The image information processing apparatus described in 1. 44. The transform coding means comprises local pattern extracting means for extracting local patterns of different scales in the vicinity of the detected feature points, and the quantizing means is a vector for each local pattern of each scale. Codebook creating means for creating a codebook of each scale that is quantized to form a representative local pattern vector of each scale, and based on the codebook group, the input signal is represented by a feature point position and a representative point corresponding thereto. 38. The image information processing apparatus according to claim 37, further comprising: a coding unit that codes the local pattern vector number. 45. The image information processing apparatus according to claim 41, wherein the neural network receives quantized data as an input signal and outputs a correction amount of an input parameter. 46. The knowledge acquisition means comprises: a voice signal input means; and means for calculating a temporal correlation of a voice signal input as knowledge data from the outside. Image information processing device. 47. The knowledge acquiring means comprises: a wireless signal receiving means; and means for calculating a temporal correlation of a wireless signal received as external knowledge data. Image information processing device. 48. The knowledge acquisition means comprises: a wired signal receiving means; and means for calculating a temporal correlation of a wired signal received as external knowledge data. Image information processing device. 49. Tool means for performing work, and tool control means for controlling the tool means based on the quantized data and the correlation.
7. The image information processing device according to 7. 50. The image information according to claim 37, further comprising audio output means for outputting an audio signal, and audio output control means for controlling the audio output means based on the quantized data and the correlation. Processing equipment.