JP2003042718A - Apparatus and method for image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor by which the behavior of a moving body is recognized with high accuracy on the basis of image information on an external environment acquired when the moving body behaves. SOLUTION: A behavior-command output part 12 outputs a behavior command which makes the moving body 32 behave. A local-feature extraction part 16 extracts local feature information on an image on the basis of the image information on the external environment acquired in the moving body 32 when the behavior command is output. A hole-feature extraction part 18 extracts feature information on a whole region in the image from the local feature information. In a learning part 20, a probability statistical model used to recognize the behavior command given to the moving body 22 is calculated on the basis of the feature information on the whole region. In a movement after that, the probability statistical model is applied to the image information on the external environmental acquired in the moving body 32, and the behavior of the moving body 32 is recognized at high speed and with high accuracy.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method, and more specifically, it accurately recognizes the behavior of a moving body using image information of the external environment acquired when the moving body moves. And an image processing apparatus and method therefor.

[0002]

2. Description of the Related Art Conventionally, as a method of recognizing the behavior of a moving body using image information of the external environment acquired by the moving body, an optical flow is detected by calculating a gradient change in image density from input continuous images. It is well known how to do it.

For example, Japanese Unexamined Patent Publication No. 2000-171250 discloses a method of detecting the current position of a moving body by using an optical flow. According to this method
When the moving body is run along a predetermined running area in advance,
The optical flow of the scene in the vicinity of the traveling area is detected at predetermined distance intervals in the traveling area, and the relationship between the optical flow and the detected position is stored. Then, the optical flow in the traveling area is also detected during the subsequent traveling, and matching with all the stored optical flows is performed. In this matching, the optical flow showing the maximum matching result is selected, and the position associated with the selected optical flow is recognized as the current traveling position of the mobile body.

Further, Japanese Laid-Open Patent Publication No. 11-134504 discloses a technique of calculating an optical flow from a moving image and processing the optical flow in a neural network layer to recognize an action and determine necessary processing. There is. According to this technique, it is possible to determine the approach of an obstacle from a moving image using a neural network having a simple structure.

[0005]

However, in order to recognize the position by the former method, it is necessary to move the predetermined region in advance and store the relationship between the optical flow and the position. In addition, in general, there are the following problems in performing feature extraction such as optical flow based on only moving images and recognizing positions and actions. That is, since the positional relationship between the light source such as sunlight or a fluorescent lamp and the camera mounted on the moving body changes every moment as the moving body moves, the image intensity such as the brightness changes and the feature extraction is performed accurately. Becomes difficult. Further, since the vibration of the moving body during movement is transmitted to the camera, the acquired continuous images are affected by the vibration and the accuracy of feature extraction is reduced. Furthermore, in order to eliminate the adverse effects of the change in the image intensity and the vibration, if the smoothing process is performed on the image information over a plurality of frames, the calculation load increases, and the target for high-speed operation with large time fluctuation There is also a problem that feature extraction for capturing an object becomes difficult.

The present invention has been made in view of the above points, and it is possible to recognize the action of a mobile body at high speed and with high accuracy by using image information of the external environment acquired by the mobile body even in an actual environment. An object of the present invention is to provide an image processing apparatus and an image processing method that can be performed.

[0007]

The principle of the present invention is to generate a probability statistical model by learning the relationship between an action command for a moving body and the image information of the external environment acquired by the moving body when moving according to the action command. The point is that the current behavior is recognized at high speed and with high accuracy based on the image information of the external environment acquired by the moving body and the probability statistical model during the subsequent movement.

According to the present invention, online learning in a real environment is performed at a pre-learning stage without performing a smoothing process or the like before performing the feature extraction required for removing the influence of noise in the conventional method. By using such noise as feature extraction data,
It aims to improve the robustness against environmental changes and avoid the problem of defective setting.

According to the present invention, an action command output section for outputting an action command for moving a moving body, and feature information of a local region of an image are extracted from image information of an external environment acquired by the moving body when the moving command is output. A local feature extraction unit that extracts the feature information of the entire region of the image using the extracted feature information of the local region, and recognizes the action command based on the feature information of the extracted entire region Provided is an image processing device including a learning unit that calculates a probability statistical model for performing.

The local feature extraction unit extracts the feature information of the local region of the image using the Gabor filter. The image intensity obtained by applying the Gabor filter of the positive component and the negative component is used for the extraction of the characteristic information of the local region. The Gabor filter is preferably applied in eight directions.

The overall feature extraction unit fuses the feature information of the local area using a Gaussian function.

The calculation of the probability statistical model is preferably performed by supervised learning using an expectation maximization algorithm and a neural network, but other learning algorithms can be used.

When the probabilistic model is formed, the behavior of the moving body can be recognized with high accuracy by applying the probabilistic model to the newly acquired image. Therefore, the image processing apparatus of the present invention further applies a Bayesian rule using a probability statistical model to image information, and further includes an action recognition unit that recognizes the action of the moving object by calculating the certainty factor for each action command. Including.

Although it is possible to recognize the behavior of the moving body as described above, it is preferable to always recognize the behavior having a certainty level or higher. Therefore, the image processing device of the present invention, the behavior evaluation unit that evaluates the behavior recognition by comparing the value based on the certainty factor calculated by the behavior recognition unit and a predetermined value, and the probability statistics according to the evaluation in the behavior evaluation unit. It may further include an attention generation unit that generates an attention request for updating the model, and an attention modulation unit that changes a predetermined parameter of the overall feature extraction unit according to the attention request. In this case, the learning unit recalculates the probability statistical model after changing the parameters. Then, the action recognition unit redoes the action recognition of the moving body using this probability statistical model.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram of an image processing apparatus 10 which is an embodiment of the present invention. Image processing device 10
From the action command output unit 12, the local feature extraction unit 16, the overall feature extraction unit 18, the learning unit 20, the storage unit 22, the action recognition unit 24, the action evaluation unit 26, the attention generation unit 28, the attention modulation unit 30, and the like. Composed.

The operation of the image processing apparatus 10 includes a pre-learning stage in which learning is carried out on a correspondence relationship between an image acquired while the moving body 32 equipped with a camera or the like is made to act in an actual environment in advance and the action at that time, and a pre-learning stage. The action recognition stage includes two stages of recognizing the action of the moving body 32 from the image information newly acquired by using the knowledge learned in the stage.

At the pre-learning stage, the action command output unit 12, local feature extraction unit 16 and overall feature extraction unit 1 shown in FIG.
8, the learning unit 20 and the storage unit 22 are used. At the action recognition stage, in addition to these, the action recognition unit 24, the action evaluation unit 26, the attention generation unit 28, and the attention modulation unit 30 are also used.

First, each part used in the pre-learning stage will be described.

The action command output section 12 outputs an action command to the moving body 32. The action command is a command that causes the moving body 32 to take an action such as going straight, turning right, turning left, or the like. Although the action command output unit 12 outputs an action command in response to an instruction signal received from the outside, the action command output unit 12 may read a command sequence set in advance in time series from a memory (not shown) and output it. In addition, the moving body 3
The configuration may be such that 2 recognizes its own action based on the acquired image, determines the action to be taken next at any time, and the action command output unit 12 outputs the action command according to the determination. The output action command is sent to the moving body 32 wirelessly or by wire, and is also supplied to the overall feature extraction unit 18 to be used for generating overall feature information described later.

An image acquisition unit 14 such as a CCD camera is attached to the moving body 32. The image acquisition unit 14 acquires images I (t) of the external environment of the moving body 32 at time t at predetermined time intervals and supplies them to the local feature extraction unit 16.

The local feature extraction unit 16 extracts the feature information of the local area of the image I (t). In the present specification, the “local area” refers to each small area when the entire area of the image I (t) acquired by the image acquisition unit 14 is divided so as to have the same size. , Each local region includes a plurality of pixels. In the present embodiment, an optical flow is calculated from two temporally consecutive images I (t) and I (t + 1),
This is the feature information of each local area (hereinafter, “local feature information”).
Used to generate). The local feature information extracted by the local feature extraction unit 16 is supplied to the overall feature extraction unit 18.

The overall feature extraction unit 18 fuses all the local feature information obtained for the image I (t) to extract feature information for the entire image (hereinafter referred to as "overall feature information"). The extracted overall characteristic information is supplied to the learning unit 20.

The learning section 20 performs learning based on the overall characteristic information supplied from the overall characteristic extracting section 18 and creates a probability model described later. In this embodiment, a known learning method using an expected value maximization algorithm and a neural network is used for this learning, but other learning algorithms may be used. The probabilistic statistical model that is the learning result is stored in the storage unit 22 and is used for recognizing the action of the moving body 32 in the action recognition stage.

When the pre-learning stage ends, the learning result (probabilistic model) is applied to the newly acquired image, and the behavior of the moving body 32 can be recognized with high accuracy. Hereinafter, each unit used in the action recognition stage will be described.

Similar to the pre-learning stage, the image acquisition unit 14 acquires images I (t) of the external environment of the moving body 32 at time t at predetermined time intervals and supplies them to the action recognition unit 24.

The action recognition unit 24 applies the probability model stored in the storage unit 22 to the supplied image I (t), calculates the certainty factor for each action command, and moves the maximum one. Recognize as the action of the body 32. The calculated certainty factor is supplied to the behavior evaluation unit 26.

The behavior evaluation unit 26 calculates the log likelihood for the certainty factor of the behavior command. The attention generator 28 does nothing if the log-likelihood of the certainty factor is equal to or larger than a predetermined value. When the logarithmic likelihood of the certainty factor is less than the predetermined value, it is determined that the recognized action does not have sufficient certainty factor, and the attention request signal is generated and supplied to the attention modulation unit 30.

Upon receiving the attention request signal from the attention generation unit 28, the attention modulation unit 30 changes (modulates) a predetermined parameter value in the learning algorithm and causes the learning unit 20 to update the probability model. The learning unit 20 stores the updated probabilistic model in the storage unit 22. The action recognition unit 24
The behavior of the moving body 32 is recognized again by applying the updated probabilistic model. As a result, an action having a certainty level or higher is always recognized.

Although the image acquisition unit 14 needs to be attached to the moving body 32, the image processing apparatus 10 may be attached to the moving body 32 either integrally with the image obtaining unit 14 or separately. It may be installed in a place different from the moving body 32. The communication between the image acquisition unit 14 and the image processing apparatus 10 may be wired or wireless.

Next, the pre-learning stage will be described in detail with reference to FIGS. FIG. 2 is a flowchart showing the flow of processing in the pre-learning stage.

Action command from action command output unit 12
When the moving body 32 acts in accordance with the
The image acquisition unit 14 provided has two images that are temporally continuous.
An image is acquired (step S42). And local feature extraction
The output unit 16 determines a local feature from the image acquired by the image acquisition unit 14.
The signature information is extracted (steps S44 to S48). concrete
For each local area image in the acquired image,
Gabor filter for each local region
Image intensity E in each direction of the Gabor filter _i(x_t, y_t)
Is calculated (step S44). Image intensity E_i(x_t, y_t)
Is calculated by the following equation (1).

E _i (x _t , y _t ) = Img _(t) × Gbr _i (+) + Img _{(t + 1)} × Gbr _i (−) (1) Here, Gbr _i (+) and Gbr _i ( -) Indicates Gabor filters with positive and negative components, respectively. Further, the subscript "i" indicates the direction of the Gabor filter, and i = 1, ..., 8 in this embodiment. Img _(t) represents a local area image of the image acquired at time t, and Img _{(t + 1)} is the next time t.
4 shows a local area image of the image acquired at +1. Furthermore, (x _t , y _t ) represents the coordinates of the pixel in the local area at time t. Therefore, E _i (x _t , y _t ) represents the image intensity in the direction i in the local area.

Although the direction of the Gabor filter and the number of Gabor filters to be applied are arbitrary, in the present embodiment, an 8-direction Gabor filter that radially extends equiangularly from the center of the entire image is imitated by receptive fields of human visual function. I'm using it.

Next, the local feature extraction unit 16 executes step 4
Image intensity E in 8 directions in each local region calculated in 4
_{i (x t, y t)} (i = 1, ..., 8) from, to select the larger direction j of the most image intensity in each local region by the following equation (2) (step S46).

J = argmax _i E _i (x _t , y _t ) (2) Here, note that the direction j (j = 1, ..., 8) is different for each local region.

Subsequently, the local feature extraction unit 16 uses the following equation (3)
Maximum image intensity _{E j (x t, y t} ) by using a Gaussian function with respect to, the local characteristic information Ψ _j (x _t for each local region, y as
_t ) is calculated (step S48).

[0038]

[Equation 1]

Here, in equation (3), “μ _j ” is the average value of the image intensities E _j (x _t , y _t ) in the local area. Further, “σ _j ” denotes the variance of these image intensities E _j (x _t , y _t ). Therefore, the local feature information Ψ _j (x _t , y _t ) is the image intensity E _j (x _t , y _t ) in the direction in which the image intensity has the maximum value in each local region, represented by the probability density distribution. . The local feature information Ψ _j (x _t , y _t ) is obtained by the number equal to the number of local regions.
Note that the direction j in which _j (x _t , y _t ) is determined is different.

When the overall feature extraction unit 18 receives the action command from the action command output unit 12 and the local feature information Ψ _j (x _t , y _t ) from the local feature extraction unit 16, the image is calculated according to the following equation (4). For each direction of the maximum intensity direction j, all local feature information Ψ _j (x _t , y _t ) obtained for that direction is fused to calculate overall feature information ρ _j (χ _t | l) (step S
50).

[0041]

[Equation 2] Here, “χ _t ” means two-dimensional Cartesian coordinates by (x _t , y _t ).

The calculated overall characteristic information ρ _j (χ _t | l) is stored by classifying the action commands output by the action command output unit 12 when the original image I (t) is obtained. (Step S52). Here, "l" represents an action command. Since three action commands (straight ahead, left turn, and right turn) are used in the present embodiment, l = 1 is a straight ahead action, l = 2 is a left turn action, and l = 3 is a right turn action command action. Accordingly, the straight moving body (l = 1) a plurality of whole feature information acquired during [rho _j | a (chi _t 1), left (l = 2) a plurality of whole feature information acquired at ρ _j (χ _t | 2) and a plurality of overall characteristic information ρ _j (χ _t | acquired at the time of right turn (l = 3)
3) are stored in separate classes.

This class is the “class of attention” Ω _l . The attention class is for not only reflecting all of the new feature information in the learning result when the new feature information is presented, but for efficiently updating the learning by focusing on the specific feature information.

The number of attention classes is not limited to three, and an arbitrary number can be set in association with the number of action commands.

Since the overall feature information ρ _j (χ _t | l) is calculated in association with the action command for the image I (t) acquired at every predetermined interval, the overall characteristic in eight directions is calculated by the equation (4). A plurality of sets of characteristic information are stored separately for each action command.

3 to 5 show the original image I (t) and the local features.
Information Ψ_j(x_t, y_t) And the overall characteristic information ρ _j(χ_t| l)
It is a figure. 3 shows the moving body 32 going straight, and FIG. 4 shows a left turn.
Occasionally, Figure 5 corresponds to the images acquired during the right turn.
It

(A) of each figure is an example of the image I (t) acquired by the image acquisition unit 14. (B) of each figure is a graph of the local feature information in one direction with a Gabor filter for the entire image, and the Z axis is the local feature information Ψ.
Represents the absolute value of _j (x _t , y _t ). In this example, the entire image is 77
It is divided into x57 local regions. (C) of each figure
It is a polar map showing the overall feature information ρ _j (χ _t | l) calculated by the calculation of equation (4) from the local feature information of (b) for each application direction of the Gabor filter. In the figure (c), the numbers 1 to 8 correspond to the application direction (upward direction, upper right direction, ...) Of the Gabor filter.

Comparing the shapes appearing in the polar maps of FIGS. 3 to 5 (c), the characteristics of the action of the moving body 32 (action command l) are obtained by obtaining the overall feature information in eight directions for each image. You can see that they are being captured.

Returning to FIG. 2, after storing the overall feature information ρ _j (χ _t | l) in step S52, the learning section 20 performs learning based on the overall feature information ρ _j (χ _t | l) (step S54 ~ S
58). Specifically, the expected value maximization algorithm (EM
Algorithm) and a neural network are used to perform supervised learning to generate a probabilistic model for recognizing the behavior of the moving body 32. Hereinafter, the EM in the present embodiment
The application of supervised learning using algorithms and neural networks will be described in order.

The EM algorithm is an iterative algorithm for estimating the parameter θ which has the maximum likelihood when the observed data is incomplete data. If the mean of the observed data is μ ^l and the covariance is Σ ^l , the parameter θ is θ (μ ^l ,
Σ ^l ). In the EM algorithm, starting from a suitable initial value of the parameter θ (μ ^l , Σ ^l )
Step (Expectation step) and M step (Maximiza
The value of the parameter θ (μ ^l , Σ ^l ) is sequentially updated by repeating the two steps of (tion step).

First, in step E, the conditional expected value ψ (θ | θ ^(k) ) is calculated by the following equation (5).

[0052]

[Equation 3]

Next, in the M step, ψ is calculated by the following equation (6).
Calculate the parameters μ ^l and Σ ^l that maximize (θ | θ ^(k) ),
This is a new estimated value θ ^{(k + 1)} .

[0054]

[Equation 4]

By repeating the E step and the M step, the conditional expectation value ψ (θ | θ ^(k) ) obtained is partially differentiated with respect to θ ^(k) . Then, by setting the result of the partial differentiation as “0”, the final μ ^l and Σ ^l are calculated. The EM algorithm is well known in the art and will not be described in further detail.

By the EM algorithm, the overall feature information of each attention class Ω _l can be represented by a normal distribution (step S54).

The overall feature extraction unit 18 uses μ ^l and Σ ^l calculated for the action command l in the following equation (7), and the overall feature information ρ _j (χ _t | l) is the class Ω _{l of the} action command _l. The prior probability p- (ρ | Ω _l ) which is the probability of belonging to (step S
56).

[0058]

[Equation 5] In the above equation, N is the number of dimensions of the overall feature information ρ _j (χ _t | l).

Next, supervised learning using a neural network will be described. In this learning, the conditional probability density function p (I (t) is set for the image I (t) acquired by the image acquisition unit 14 using the attention class Ω _l as a teacher signal.
| Ω _l ) is calculated (step S58).

FIG. 6 is a diagram showing an example of the structure of a hierarchical neural network used for supervised learning using this neural network. This hierarchical neural network has three layers of nodes, the input layer 72 is the original image I (t), the intermediate layer 74 is the overall feature information ρ _j (χ _t | l), and the output layer 76 is the action command l. Corresponds to each class of attention Ω _l . Although only three nodes are drawn in the input layer 72 for simplicity, there are actually as many nodes as there are images I (t). Similarly, the intermediate layer 74 has the same number of nodes as the total feature information ρ _j (χ _t | l) as the input layer 72, and both nodes are connected in a one-to-one relationship. The number of nodes in the output layer 76 is the same as the number of attention classes Ω _l (three in this embodiment).

In FIG. 6, “λ” is the synapse weight of the hierarchical neural network. Whole feature information by EM algorithm ρ _j (χ _t | l) has been required probability belonging to the class Omega _l of each note, also the whole feature information ρ _j (χ _t | l) is the set of images I ( Since t) and I (t + 1) are calculated in a one-to-one correspondence, the image I (t) and the attention class can be obtained by repeating supervised learning with the attention class Ω _l as the teacher signal. The probabilistic relationship of Ω _l (that is, λ in FIG. 6) is determined. This probabilistic relationship is the conditional probability density function p (I (t) | Ω _l ). Hierarchical neural networks are well known in the art and will not be described in further detail.

By the supervised learning using such a neural network, the image I (t) and the attention class Ω _l
Conditional probability density function p (I
(t) | Ω _l ) can be obtained.

The processing of steps S54 to S58 is as follows.
It is executed for each action command l. Therefore, in the present embodiment, the a priori probability p- (ρ | Ω _l ) and the conditional probability density function p (I (t) | Ω _l ) for each of the action commands l = 1, 2, and 3.
(These are collectively referred to as a “probabilistic model”) is calculated.

The probability model calculated by the learning unit 20 is stored in the storage unit 22 (step S60). In the case of continuing the pre-learning, the result in step S62 is "yes", and the series of processes from step 42 is repeated again, and the probabilistic model is updated. The pre-learning is executed for all the images I (t) acquired at a predetermined interval while the moving body 32 is acting. Then, the process ends for a predetermined number of images I (t), etc., and ends when it is determined that a stochastic model with sufficient accuracy for action recognition has been generated (step S64).

Next, the action recognition stage will be described in detail with reference to FIGS. FIG. 7 is a flowchart showing the flow of processing in the action recognition stage.

The image acquisition unit 14 acquires two images at predetermined time intervals (step 82).

Next, the probability model calculated at the time of pre-training, that is, the prior probability p- (ρ ^l | Ω _l ) and the conditional probability density function p (I (t) | Ω _l ) are used in the following Bayes law. Is
The confidence level p (Ω _l (t)) (confidence) of each attention class Ω _l is calculated (step S84). This certainty factor p (Ω _l (t))
Represents the probability that the image I (t) acquired by the image acquisition unit 14 belongs to each attention class Ω _l .

[0068]

[Equation 6] Then, the calculated three confidences p (Ω ₁ (t)), p (Ω
_{Of 2} (t)) and p (Ω ₃ (t)), the maximum one is selected (step S86).

The behavior evaluation unit 26, with respect to the certainty factor p (Ω _l (t)) selected by the action recognition unit 24, the log likelihood log p.
It is determined whether (Ω _l (t)) is larger than a predetermined value K (step S88). When log p (Ω _l (t))> K, the action command l corresponding to the attention class Ω _l with the highest certainty factor
Is recognized as the action of the moving body 32 that is actually performed when the image I (t) is acquired (step S9).
2).

On the other hand, if log p (Ω _l (t)) ≦ K, the caution generator 28 issues a caution request. Note The modulation unit 30 is calculated by the formula (7).
The Gaussian mixture "m" in (1) is increased by a predetermined value (that is, attention modulation) (step S90). Then, in the learning unit 20, steps S56 to S60 of FIG.
The series of processing of is executed again, and the probabilistic model (prior probability p-
(ρ | Ω _l ) and the conditional probability density function p (I (t) | Ω _l )) are updated.

Then, the process returns to step S84 in FIG. 7 and the processes of steps S84 to S88 are repeated, and the Gaussian mixture m increases until the logarithmic likelihood log p (Ω _l (t)) becomes equal to or larger than the predetermined value K. To be done.

Note that a probabilistic model created in advance may always be used without such an updating process.

As described above, according to the present invention, the action of the moving body is not recognized only from the image information, but the relationship between the overall feature information extracted from the image information and the action command is learned in advance, Since the action recognition is performed using the learning result, the action of the moving body can be recognized at high speed and with high accuracy even in the actual environment.

Even if the moving body 32 does not move correctly in response to a given action command due to a tire mounting failure or the like, the true moving state can be grasped from the image.

Next, examples of the present invention will be described. FIG. 8 is a radio control car (hereinafter referred to as “RC car”) 10 equipped with the image processing apparatus 10 of the present invention.
It is a block diagram of 0. Each part of the image processing apparatus 10 is shown in FIG.
It has a function similar to that described in connection with. In addition to the image processing device 10, the RC car 100 includes an image input camera 114 for acquiring an image, a steering control unit 132 and a drive control unit 134 for controlling the behavior of the RC car 100 in accordance with a behavior command, and communication with the outside. A receiver 136 and a transmitter 138 are provided. The receiver 136 receives the action command command signal from the outside, and the action command output unit 1
Supply to 2. The RC car 100 takes one of three actions of going straight, turning left, and turning right according to the action command. Also,
The action of the RC car 100 recognized by the action recognition unit 24 is transmitted to the outside via the transmitter 138.

Turn right on this RC car 100, go straight,
The following is a description of the results when the action recognition is performed on the image of 24 frames after the image is acquired while the action commands of turning left and the left turn are given and the pre-learning stage is completed.

FIG. 9 shows the log-likelihood of the certainty factor of the attention class.
6 is a graph showing changes in the value of log p (Ω _l (t)). The horizontal axis represents the value of the Gaussian mixture m in equation (7), and the vertical axis represents the logarithmic likelihood log p (Ω _l (t)) of the certainty factor. It can be seen from FIG. 9 that the log-likelihood is saturated when the number of Gaussian mixtures is around 50. The fact that the log-likelihood of the confidence level of the attention class for each image is large means that it is highly likely that the RC car 100 at the time of acquiring each image is performing an action corresponding to the attention class. Processor 1
This corresponds to 0 being recognized.

FIG. 10 shows the result of the logarithmic likelihood log p (Ω _l ) of the certainty factor obtained by the equation (8) when the Gaussian mixture (the number of Gaussian functions) m is m = 20, and FIG. 11 shows m. = 50
The same result is shown for. The vertical axis in FIGS. 10 and 11 is, in (a), the logarithmic likelihood log p (Ω ₁ ) of the certainty factor for the attention class Ω ₁ of the action command l = 1 (straight ahead), (b)
In action command l = 2 log-likelihood confidence for attention class Omega ₂ of (left) log p (Ω _2), for the attention of the class Omega ₃ (c), the action command l = 3 (right turn) The log-likelihood log p (Ω ₃ ) of the certainty factor is shown. The abscissa of each figure corresponds to 24 images for which action recognition is performed. The first 8 out of 24 images (images 1-8) are R
When the left turn action command l = 2 is given to the C-car 100, the middle eight (images 9 to 16) are the last eight (images 17 to 24) when the straight ahead action command l = 1 is given. )
Correspond to the right turn action command l = 3.

Therefore, referring to FIG. 10, (a) shows the log-likelihood of the maximum confidence for the central eight images (that is, when going straight), and (b) shows the first eight images (that is, when turning left). (C) is the same for each of the last eight images (that is, when turning right). However, the variation of log-likelihood between images is large, and it cannot be said that the recognition of actions is sufficient.

Next, referring to FIG. 11, (a), (b), (c)
In both cases, as in FIG. 10, the image corresponding to the action command 1 shows the log likelihood of the maximum certainty factor. However, in FIG. 11, there is less variation in the log likelihood between images as compared with FIG. 10, and the image is smooth. This was achieved by increasing the Gaussian mixture by attentional modulation.

As described above, by using the image processing device of the present invention, the reliability is improved by the learning of the attention class formed in the bottom-up in the pre-learning stage, and the confidence is increased in the action recognition stage. Since the probability model is updated until the log-likelihood of the degree exceeds a predetermined value, the action recognition accuracy is improved.

[0082]

According to the present invention, the action of the moving body is not recognized only from the image information, but the relationship between the image information and the action command is learned in advance, and the learning result is used to make a decision. Therefore, the behavior of the moving body can be recognized at high speed and with high accuracy even in the actual environment.

[Brief description of drawings]

FIG. 1 is a functional block diagram of an image processing apparatus that is an embodiment of the present invention.

FIG. 2 is a flowchart showing a pre-learning stage of the image processing method according to the present invention.

FIG. 3 is a diagram showing an image recognition result when a moving body is straight ahead (Ω ₁ ).

FIG. 4 is a diagram showing an image recognition result when a moving body turns left (Ω ₂ ).

FIG. 5 is a diagram showing an image recognition result when a moving body turns right (Ω ₃ ).

FIG. 6 is a diagram showing a configuration example of a hierarchical neural network used for supervised learning using a neural network.

FIG. 7 is a flowchart showing an action recognition step of the image processing method according to the present invention.

FIG. 8 is a block diagram showing the overall configuration of an RC car using the image processing apparatus according to the present invention.

FIG. 9 is a diagram showing a change in log likelihood of confidence.

FIG. 10: RC when Gaussian mixture m = 20
It is a figure which shows the recognition result of the behavior of a car.

FIG. 11: RC when Gaussian mixture m = 50
It is a figure which shows the recognition result of the behavior of a car.

[Explanation of symbols]

10 Image processing device 12 Action command output section 14 Image acquisition unit 16 Local Feature Extraction Unit 18 Overall Feature Extraction Unit 20 Learning Department 22 Memory 24 Action recognition unit 26 Behavior Evaluation Department 28 Attention generator 30 Attention modulation section 32 moving bodies

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G06T 7/20 100 G06T 7/20 100 // G05D 1/02 G05D 1/02 K (72) Inventor Akatsuka Koji 1-4-1 Chuo, Wako City, Saitama Prefecture F-term in Honda R & D Co., Ltd. (Reference) 2F065 AA01 BB15 DD06 FF04 JJ03 JJ26 PP01 QQ00 QQ14 QQ17 QQ24 QQ26 QQ27 QQ33 QQ41 5B057 AA05 AA06 BA02 DB06 DC07 DB07 CE07 DA07 DB07 DB07 DC34 DC40 5H301 AA01 DD02 GG09 5L096 AA06 BA04 BA05 CA04 FA34 FA67 FA69 GA30 GA55 HA04 HA11 JA18 KA04 KA15

Claims (16)

[Claims] 1. A behavior command output unit for outputting a behavior command for acting a moving body, and a local for extracting characteristic information of a local region of an image from image information of an external environment acquired in the moving body at the time of outputting the behavior command. A feature extraction unit, which extracts feature information of the entire region of the image using the extracted feature information of the local region, and an action command to recognize the action command based on the feature information of the extracted entire region An image processing apparatus including: a learning unit that calculates a probability statistical model of. 2. The image processing apparatus according to claim 1, wherein the local feature extracting section extracts feature information of a local region of an image using a Gabor filter. 3. The image processing apparatus according to claim 2, wherein the image intensity obtained by applying a Gabor filter of a positive component and a negative component is used for extracting the characteristic information of the local region of the image. 4. The image processing apparatus according to claim 2, wherein the Gabor filter is an 8-direction Gabor filter. 5. The image processing apparatus according to claim 1, wherein the overall feature extraction unit fuses feature information of local areas using a Gaussian function. 6. The image processing apparatus according to claim 1, wherein the probability statistical model is generated by supervised learning using an expected value maximizing algorithm and a neural network. 7. An action recognition unit for recognizing the action of the moving body by applying Bayes' rule using the probability statistical model to the image information and calculating a certainty factor for each action command. The image processing apparatus according to claim 1. 8. An action evaluation unit that evaluates the action recognition by comparing a value based on a certainty factor calculated by the action recognition unit with a predetermined value, and the probability statistics according to the evaluation by the action evaluation unit. An attention generating unit that generates an attention request for updating a model, and an attention modulation unit that changes a predetermined parameter of the overall feature extraction unit according to the attention request are further provided, and the learning unit re-executes the parameter after changing the parameter. The image processing apparatus according to claim 7, which calculates a probability statistical model. 9. An action command for moving a moving body is output, feature information of a local region of an image is extracted from image information of an external environment acquired in the moving body at the time of outputting the action command, and the extracted local region is extracted. An image processing method comprising: extracting feature information of an entire region of an image using the feature information of, and calculating a probability statistical model for recognizing an action command based on the extracted feature information of the entire region. 10. The image processing method according to claim 9, wherein the extraction of the characteristic information of the local region is performed using a Gabor filter. 11. The image processing method according to claim 10, wherein image intensity obtained by applying a Gabor filter of a positive component and a negative component is used for extracting the characteristic information of the local region. 12. The image processing method according to claim 10, wherein the Gabor filter is an 8-direction Gabor filter. 13. The image processing method according to claim 9, wherein the feature information of the entire region is extracted by fusing feature information of the local region using a Gaussian function. 14. The image processing method according to claim 9, wherein the probability statistical model is generated by supervised learning using an expected value maximizing algorithm and a neural network. 15. The method further comprises recognizing the action of the moving body by applying a Bayes rule using the probability statistical model to the image information and calculating a certainty factor for each action command. Any one of items 9 to 14
The image processing method described in the item. 16. The behavior recognition is evaluated by comparing a value based on the calculated certainty factor with a predetermined value, and a caution request for updating the probability statistical model is generated according to the evaluation. The image processing method according to claim 15, further comprising changing a predetermined parameter of the overall feature extraction unit according to a request, wherein the probability statistical model is recalculated after the parameter is changed.