JPH08202877A - Picture recognition device - Google Patents

Abstract

PURPOSE: To speedily and accurately recognize desired objects such as white line from a picked-up picture obtained under various conditions by executing correlation arithmetic between a reference picture and the picked-up picture through the use of the neural network(NN) of excellent panning ability. CONSTITUTION: A vehicle is provided with a picture pickup means 1 to pickup a picture anterior to the vehicle including the white line on the surface of a road by the picture pickup means 1. In addition, a reference picture storing means 2 is provided to store a reference picture specified in advance as a pattern. The reference picture is inputted to an arithmetic means 3 consisting of NN to learn NN by a pattern learning means 4. Next, with respect to NN learned through the use of the reference picture, the picture picked-up by the picture pickup means 1 (a processing object picture) is inputted and processed through the use of NN to output a correlation value. Based on the change of this correlation value corresponding to the position, an object such as the white line is detected. As the result of this, even when the object varies in terms of the shape and luminance, the object can speedily and accurately be recognized from the obtained picture.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image recognition device, and more particularly to a device using a neural network (hereinafter referred to as NN).

[0002]

2. Description of the Related Art Conventionally, various devices have been proposed for reducing the burden on a vehicle driver, and the ultimate one is an automatic driving device that recognizes a running shape and performs automatic steering. As a method of recognizing the shape of the road, a method of recognizing the road by image processing and an electromagnetic induction method have been proposed. Among them, the method of recognizing the white line on the road surface by image processing can use the existing white line as it is. Because of its advantages, research and development are being carried out earnestly toward its realization.

For example, in Japanese Patent Laid-Open No. 6-119593, an image obtained by a camera is stored in a frame memory,
An apparatus has been proposed which reads out from a memory for each predetermined search window, compares predetermined templates, and performs a correlation calculation. The position with a high correlation with the template is the white line position, and the same correlation calculation is performed while sequentially moving the search window from the side closer to the vehicle to the side farther from the vehicle, and continuous white line positions are detected.

[0004]

However, the correlation method based on the difference sum using the template lacks flexibility and responsiveness to the shape change and brightness change of the target object such as a white line, and the correlation amount due to these influences There was a problem that the decrease was remarkable.

Of course, in order to solve such a problem, a plurality of templates are prepared in advance and a plurality of correlation calculations are performed, or the templates are sequentially updated to the latest one according to the change of the target object. An approach such as performing a correlation operation after performing normalization can be considered.
A new problem arises that the processing becomes complicated and the processing time increases. In particular, when a vehicle is mounted on a vehicle and white line detection is performed in real time, high speed is highly required, and thus such an increase in processing time cannot be ignored.

The present invention has been made in view of the above problems of the prior art. Image recognition can be quickly and reliably recognized from an acquired image even if a target object such as a white line changes in various points in terms of shape and brightness. To provide a device.

[0007]

In order to achieve the above object, the image recognition apparatus according to claim 1 stores an image pickup means for picking up an image of a vehicle surroundings and a representative reference image prepared in advance. It is characterized by comprising a storage means and a calculation means for calculating a correlation between an image in a predetermined region of the image obtained by the image capturing means and the reference image, by NN learned using the reference image.

In order to achieve the above object, the image recognition apparatus according to a second aspect is the image recognition apparatus according to the first aspect, wherein a plurality of the reference images are prepared according to the image pickup conditions of the image pickup means. The calculation means calculates the correlation using the NN additionally learned using the plurality of reference images.

[0009]

In the image recognition apparatus according to the first aspect, the correlation calculation between the reference image and the picked-up image is performed by using the NN having excellent generalization ability. The NN learning is performed using the reference image of the object to be recognized, whereby the target object can be surely recognized from the images obtained under various conditions without adding complicated processing.

In the image recognition apparatus according to the second aspect of the present invention, the additional learning ability of the NN is skillfully utilized to prepare a plurality of reference images to perform the additional learning of the NN. By this additional learning, the generalization ability of the NN is expanded, and the target object can be reliably recognized even in a captured image under more severe conditions.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings by exemplifying a case of recognizing a white line on a road surface.

FIG. 1 is a conceptual block diagram of this embodiment. The image pickup means 1 is provided in the vehicle, and the image pickup means 1 picks up a vehicle front image including a white line on the road surface. Further, a reference image storage means 2 is provided, and a predetermined reference image is stored as a pattern. Reference image is N
It is input to the calculation means 3 composed of N, and the pattern learning means 4 learns NN. This learning is performed by determining the weight of each neuron forming the NN so that a predetermined value (for example, 1.0) is output from the output layer when the reference image pattern is input to the NN input layer. However, the details will be described later. The image (processing target image) captured by the image capturing unit 1 is input to the learned NN using the reference image, processed using the NN, and the correlation value is output. This correlation value corresponds to the correlation value obtained by the conventional sum of differences, and a target object such as a white line is detected based on the change in the correlation value depending on the position.

Further, the pattern learning means 4 carries out additional learning of the NN using a plurality of reference images stored in the reference image storage means 2 when additional learning is required according to the image pickup conditions, and the input image pickup is carried out. Calculate the correlation value of the image. Note that the additional learning means that there are two reference images (reference images I and II, respectively).
If it exists, the reference image I is first used to generate N
After learning N, the reference image I is further learned with respect to the learned NN.
This means learning using I, and does not mean that the reference images I and II are simultaneously input to the NN input layer to perform learning.

FIG. 2 shows a concrete block diagram of the present embodiment. A CCD camera 10 is provided in the vehicle as the image capturing means 1 and captures a vehicle front image. The obtained forward image is output to the A / D converter 12, converted into a digital signal, and stored in the video RAM 14. In the present embodiment, the captured image is composed of 512 × 512 pixels, and each pixel is digitized in the A / D converter 12 into 8-bit 256 gradations.

On the other hand, a standard image of 8 × 8 pixels prepared in advance is stored in the main RAM 18, and the standard image is N
It is supplied to a DSP (digital signal processor) 16 as a calculation means 3 that configures N as a hardware. DSP
In No. 16, the weight of each neuron constituting the NN is determined by learning according to the NN software stored in the user ROM so that a predetermined value is output from the output layer when the reference image is input to the NN input layer. To do. Therefore, this user ROM 22 constitutes the pattern learning means 4. The NN model used is a perceptron type NN.

After the learning is completed, the images in a predetermined area among the images stored in the video RAM 14 are read out in a predetermined order and supplied to the DSP 16. The DSP 16 performs an operation using the input image and the learned NN, and outputs the operation result to the main RAM 18. The image sequentially supplied to the DSP 16 is an 8 × 8 pixel image corresponding to the size of the reference image. After the calculation is performed on all the images in the predetermined area, the DSP 16 determines the white line position from the calculation result stored in the main RAM 18, and stores it in the video RAM 24 together with the captured image. The data stored in the video RAM 24 is converted into an analog signal by the D / A converter 26, output to the video display 28, and the white line position is displayed on the screen together with the captured image.

Here, the number of input nodes of the NN formed by the DSP 16 is set to 64 in order to allocate one node to each pixel of the input image of 8 × 8 pixels, and the number of intermediate nodes is in the range of 2 to 32. The number of output nodes is set to one or two according to two types of learning methods described later.

FIG. 3 shows a search area which is a predetermined area in the picked-up image stored in the video RAM 14 and an input image which is sequentially input to the DSP 16 which constitutes the NN. The search area is composed of 8 × L pixels, and the input pixels are 8 × 8 pixels as described above. The input image sequentially shifts laterally by one pixel in the figure, and the DSP 16 executes the calculation by NN for each position.

When all images in the search area are input to the DSP 16 in this manner, NN outputs are obtained for all positions in the search area, and eventually NN outputs are obtained as a function of the position. Become.

FIG. 4 shows an example of such an NN output, and shows the NN output for each image position X, that is, the correlation value between the input image and the reference image. This relationship corresponds to the correlation value based on the conventional sum of differences. Therefore, it is possible to specify which position has the highest correlation and is the white line position from the absolute value and the slope of this graph.

FIG. 5 shows the position of the white line used when learning the NN, which is the left edge portion of the standard white line. By inputting the image of this portion to the NN as a reference image, the standard luminance pattern of the left edge of the white line is learned. Note that the pattern of the edge portion is learned in order to perform recognition that does not depend on the width of the white line, and by inverting the luminance distribution, it is possible to easily extend the recognition to the edge on the right side of the white line.

<NN Learning Method> Here, the NN learning method will be described. The purpose of learning the NN in this embodiment is to enable the NN to recognize a white line candidate position under various imaging conditions by learning the typical reference image pattern by the NN itself. . Therefore, in this embodiment, learning is performed by the following two methods.

(1) Learning method I for learning the reference image pattern during daytime fine weather (2) Learning method II for learning by adding the reference image pattern during daytime cloudy weather to the pattern of (1).

FIG. 6 shows a reference image pattern used in the learning method I. In the learning method I, three reference image patterns (a), (b), and (c) are prepared,
(B) corresponds to the reference image of the left edge portion of the white line during fine weather in the daytime. Then, the learning method outputs 1 for the pattern (b),
Learning is performed so that the output becomes 0 for the patterns (a) and (c). Further, FIG. 7 shows a reference image pattern used in the learning method II. Four reference image patterns (a), (b), (c), (d) are prepared,
(A), (b), and (d) are the same as the learning method I, and the newly added pattern (c) corresponds to the reference image of the left edge portion of the white line during cloudy daytime. Then, the number of output nodes of the NN is set to 2, and outputs 00 to the patterns (a) and (d)
Learning is performed so that the output is 10 for the pattern (b) and the output 01 is for the pattern (c).

FIGS. 8A and 8B show a learning method I pattern (b) and a learning method II pattern (c), respectively.
An example of the actual luminance distribution of the input image used in is shown. As described above, the brightness is represented by 0 to 255, and NN
Learning is performed using values that are obtained by dividing these values by 255 and normalizing them into a range of 0 to 1. Learning methods I and II
Of the learning method I and the pattern (d) of the learning method II are all 255.
It is represented by (1 when normalized).

FIG. 9 shows a NN learning flowchart executed by the DSP 16 based on the NN software. This learning algorithm is based on the back propagation (BP) method. First, the coupling coefficient and offset are initialized using random numbers (S101), and a learning pattern is set (S102). That is, the input patterns shown in FIGS. 6 and 7 and the output patterns corresponding to them are set. The learning method I has three learning patterns, and the learning method II has four learning patterns. Next, the output calculation of the middle layer unit and the output calculation of the output layer unit are performed (S103, S1).
04). This calculation calculates the input to the k-th layer i-th unit by i ^k
_i , the output is O ^k _i , k from the i-th unit of the (k−1) th layer
If the weight for the j-th unit of the layer is W ^kk-1 _ij ,

## EQU1 ## i ^k _j = Σ (W ^{k-1, k} _ij O ^k-1 _i ), O ^k _i =
f (i ^k _i ) where Q is an offset

## EQU00002 ## f (x) = 1 / {1 + exp (-x + Q)}.

After the output calculation of the output layer unit is completed, the error is calculated, and further the error of the intermediate layer unit is calculated (S105, S106). In this error calculation, if the i-th element of the presented output pattern m is t ^m _i and the i-th element of the output pattern m actually output by the NN is O ^m _i ,

(3) E = Σ {1 / 2Σ (t ^m _i −O ^m _i ) ² }. Then, the coupling coefficient between the intermediate layer and the output layer and the offset of the output layer unit are updated according to the value of the error E (S107), and the coupling coefficient between the input layer and the intermediate layer and the offset of the intermediate layer unit are updated. Perform (S108). After the learning with one set of input / output patterns is completed in this way, the learning with the next input / output pattern is performed (S109, S110). After the learning in all patterns is completed (learning method I has 3 sets, learning method II has 4 sets), the number of learning iterations is sequentially incremented (S111), and these are repeated until the predetermined number of iterations is reached. The process is repeated (S112).

It should be noted that which of the learning method I and the learning method II is to be used can be appropriately selected. When the condition is favorable, the learning method I is selected, and when the condition is relatively severe, the learning method II is selected. Can be selected. Alternatively, 2 of those learned in Learning Method I and those learned in Learning Method II in advance.
Consider a two-step process in which types are prepared and image recognition is first performed by the learning method 1, and when a good result is not obtained, then image recognition is performed by the learning method II. To be

<White Line Recognition> When the learning of the NN is completed as described above, the process shifts to the evaluation of the image actually obtained by the CCD camera 10. FIG. 10 shows a DSP whose learning has been completed.
The white line recognition processing flowchart in 16 is shown.
It should be noted that FIG. 10 is a processing flowchart when learning is performed by either the learning method I or the learning method II. First, as shown in FIG. 3, after setting a search area in the image (S201) and resetting the position X of the input image area (S202), each pixel of the 8 × 8 image is input to 64 input nodes of NN. Input (S203). NN (actually D
In SP16), the calculation is performed through the input layer, the intermediate layer, and the output layer, and finally a numerical value between 0 and 1 is output from one output layer (S204). Of course, when the input image pattern is close to FIG. 6A and FIG. 6C, the output value is close to 0, and when the input image pattern is close to FIG. 6B, the output value is close to 1. Similarly, the input image is sequentially moved in the search area and the calculation is repeated. When the input image position x reaches the end (right end) of the search area (S
205), the DSP 16 obtains the position X where the NN output value obtained as a function of the position x is the maximum, that is, the value closest to 1 (S206).

Then, it is judged whether or not the NN output value at the position X is larger than a predetermined value a, and whether or not the absolute value of the change rate (gradient of the graph) of the NN in the vicinity thereof is larger than a predetermined value b ( S207), if both are large,
The position having the strongest correlation is X, and it is determined that the position X has a white line (S208). On the other hand, if the NN output value is less than or equal to the predetermined value or the change rate is less than or equal to the predetermined value, it is determined that the correlation is not sufficiently strong and there is no white line (S20).
9).

Since the number of output nodes is two when the learning method II is used, the larger one of the two output values is set as the NN output value at the position x in S206. When both the learning method I and the learning method II are used, the NN output is first calculated by the learning method I and the NN output value is below a predetermined value or the change rate is below a predetermined value in S207. If it is determined that the NN output is calculated again by the learning method II, the determination of S207 is performed. As a result, when the NN output value at the position X is larger than the predetermined value a and the absolute value of the change rate (gradient of the graph) of the NN in the vicinity thereof is larger than the predetermined value, it is determined that the position having the strongest correlation is X. It may be determined that the position X has a white line, and if not, it may be determined that there is no white line.

11 to 15 show the luminance distributions of five images for which white line recognition was attempted in this embodiment. In each figure, the horizontal axis represents the horizontal position in the image, and the vertical axis represents the brightness of the image slice (0 to 25
5) is shown. FIG. 11 is an image in the case where a shadow of a pedestrian bridge exists on a part of the white line. In the area where the shadow exists, the brightness difference between the road surface and the white line is small, the brightness inside the white line is constant, and the edge of the white line is clear. It has the feature of being. FIG. 12 shows an image in a tunnel, which is characterized in that the difference in brightness between the road surface and the white line is small, the brightness in the white line is constant, and the edges of the white line are somewhat unclear. FIG. 13 is an image with a faint white line. The brightness difference between the road surface and the white line is small, the brightness inside the white line is not constant, and the edges of the white line are somewhat unclear. FIG. 14 is an image in the case of rain, and has the characteristics that the difference in brightness between the road surface and the white line is small, the brightness within the white line is not constant, and the edges of the white line are somewhat unclear. FIG. 15 is an image at night, and has a feature that the brightness difference between the road surface and the white line is large, the brightness inside the white line is not constant, and the edge of the white line is unclear.

FIGS. 16 to 21 show the DSP when each image having the luminance distribution shown in these figures is input.
16 is the processing result (NN output value) of 16. Note that FIG. 16 shows a processing result by the learning method I when an image in a standard condition (there is no shadow in the clear sky and no white line blurring) is input, and the horizontal axis represents the position x and the vertical axis represents the NN of 0 to 1. It represents the output value (correlation amount). Further, FIG. 17 shows the NN output value when the image of FIG. 11 is input, and the value when the learning method I is used for learning. FIG. 18 shows NN output values when the image of FIG. 12 is input, and values when learning is performed by the learning method II. The learning method II is used because the learning method I did not give good results. FIG. 19 shows the NN output value when the image of FIG. 13 is input, and the value when the learning method II is used for learning. FIG. 20 shows NN output values when the image of FIG. 14 is input, and values when learning is performed by the learning method II. Figure 2
1 is the NN output value when the image of FIG. 15 is input,
It is a value when learning is performed by the learning method I. In addition, in each figure, the correlation amount obtained by the correlation calculation method by the sum of differences using the conventional template is also shown for comparison. Compared to the conventional difference calculation, the correlation amount by NN is large, and
It can be seen that it shows a sharp peak.

<Evaluation> The following criteria were used to evaluate the image processing results of FIGS. 16 to 21. (1) N
Processing by N (a) When the NN output at a position other than the white line is 0.5 or more or the NN output at a white line position is 0.5 or less: Evaluation X (b) The NN output at a position other than the white line is 0. 5 or less, and the NN output at the position of the white line is 0.5 to 0.8: Evaluation Δ (c) The NN output at positions other than the white line is 0.5 or less, and the NN output at the position of the white line is 0. 8 or more: Evaluation ○ (2) Processing by sum of differences using conventional template Evaluation was performed in the same manner as in the case of NN with reference to the correlation amount at the white line position and the slope of the correlation graph.

FIG. 22 shows the result of evaluation based on such criteria. Image numbers 1 to 6 correspond to FIGS. 16 to 21, respectively. In the figure, “2” and “32” in the NN learning methods I and II represent the number of nodes in the intermediate layer. From these results, it can be seen that the output difference between the white line and the road surface appears more clearly in the processing method using the NN than in the conventional method, and sufficient white line recognition can be performed.

It should be noted that the difference in brightness between the white line and the road surface has a great influence on the recognition ability, and the image numbers 3, 4, 5 having a small difference are recognized.
In (corresponding to FIG. 18, FIG. 19, and FIG. 20), the learning method I could not obtain a sufficient recognition result as described above. Moreover, even if the number of nodes in the middle layer was changed, there was no big difference in the recognition results, and the same recognition result was obtained in all cases.

As described above, in the present embodiment, the captured image is processed by using the NN, so that the white line cannot be easily recognized by the conventional correlation calculation method using the sum of differences. You can recognize the white line on the. In particular, since the NN has an additional learning capability, in the case of an image with more severe conditions, there is an advantage that it can be easily dealt with by additionally learning a new white line luminance change pattern as a reference image. Further, since the additional learning is performed by so-called off-line processing (which is separately performed prior to the actual image processing), the real time for the recognition processing does not increase.

In the above embodiment, the case of recognizing the white line in the image is illustrated, but the present invention is not limited to this, and for example, the edge portion of another vehicle existing around the vehicle is detected. Alternatively, applications such as recognizing signs existing on the road surface can be considered. Needless to say, when recognizing the edge portion of another vehicle, it is necessary to learn the NN using a standard edge image pattern as a reference image.

[0039]

As described above, according to the image recognition device of the first or second aspect, a desired object such as a white line can be swiftly and surely obtained from captured images obtained under various conditions. Can be recognized.

In particular, according to the image recognition device of the second aspect, by the additional learning of the NN, it becomes possible to easily cope with an image under more severe conditions without increasing the recognition processing time. .

[Brief description of drawings]

FIG. 1 is a conceptual configuration diagram of an embodiment of the present invention.

FIG. 2 is a configuration block diagram of the embodiment.

FIG. 3 is an explanatory diagram of a search area of the same embodiment.

FIG. 4 is an explanatory diagram of a correlation amount calculation by the NN of the embodiment.

FIG. 5 is an explanatory diagram of an NN learning position of the embodiment.

FIG. 6 is an explanatory diagram of a reference image pattern of learning method I according to the embodiment.

FIG. 7 is an explanatory diagram of a reference image pattern of learning method II according to the same embodiment.

FIG. 8 is an explanatory diagram of a luminance distribution of a reference pattern of the example.

FIG. 9 is a NN learning flowchart of the embodiment.

FIG. 10 is a white line recognition flowchart by the NN of the embodiment.

FIG. 11 is an image luminance distribution diagram of a shadow of a pedestrian bridge in the same example.

FIG. 12 is an image brightness distribution diagram in the tunnel of the embodiment.

FIG. 13 is an image luminance distribution chart of a faint white line in the example.

FIG. 14 is an image luminance distribution diagram in rainy weather in the example.

FIG. 15 is a nighttime image brightness distribution diagram of the same example.

FIG. 16 is an explanatory diagram of a correlation amount in fine weather according to the same embodiment.

FIG. 17 is an explanatory diagram of a correlation amount of a pedestrian bridge shadow in the example.

FIG. 18 is an explanatory diagram of a correlation amount in the tunnel of the example.

FIG. 19 is an explanatory diagram of a white line blur correlation amount in the example.

FIG. 20 is an explanatory diagram of a correlation amount in rainy weather in the example.

FIG. 21 is an explanatory diagram of a nighttime correlation amount according to the embodiment.

FIG. 22 is a table showing the evaluation results of the same example.

[Explanation of symbols]

10 CCD camera, 12 A / D converter, 14
Video RAM, 16 DSP, 18 Main RAM, 20
System ROM, 22 User ROM, 24 Bide R
AM, 26 D / A converter, 28 video display.

Claims (2)

[Claims] 1. An image pickup means for picking up a vehicle surroundings image, a storage means for storing a representative reference image prepared in advance, an image in a predetermined region of the image obtained by the image pickup means, and the reference. An image recognizing device, comprising: a calculating unit that calculates a correlation between images by a neural network learned using the reference image. 2. The image recognition apparatus according to claim 1, wherein a plurality of the reference images are prepared according to the image pickup conditions of the image pickup means, and the calculation means additionally learns using the plurality of reference images. An image recognition device characterized by calculating a correlation using a network.