JP2915067B2 - Target area identification method in image processing - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a target area identifying method for extracting a target area by finding a boundary between a target area and a background area in image processing.

[Conventional technology]

Autonomous traveling of an automobile requires a technology for discriminating a traveling route by image processing. Conventionally, a road edge or a running course is obtained from an image division area by sequentially tracking the boundary between the obtained running course area and the background area to obtain a point sequence group corresponding to the boundary line. That is, the pixels located at the boundary of the travel road area are sequentially tracked one by one along this boundary, and the trace of this tracking is used as a point sequence, and the road edge is determined based on the point sequence. .

[Problems to be solved by the invention]

For this reason, in the conventional method for determining the boundary end of the traveling area, the number of point sequence groups to be determined increases. Therefore, it is necessary to approximate the boundary of the travel road area with a polygon and to thin out extra points, which takes time. Also, the boundary of the traveling road area obtained as a result of the image processing may be smooth,
In practice, the shape of this boundary becomes complicated. Therefore, a large amount of data can be obtained by sequentially tracking the boundary pixels in a plurality of areas, and as a result, the processing takes time. If the processing time is long as described above, the automatic running of the self-sustaining running is hindered.

[Means for solving the problem]

The present invention has been made in order to solve such a problem, and divides an image labeled in each area obtained by dividing an image by a dividing line in a certain direction, and displays a list in a direction orthogonal to the certain direction as a dividing line. Created for each area, scan the image along the dividing line, identify the changing point of the label value of each area, find the boundary point located at the intersection of the dividing line and the boundary line of each area, and list in a certain direction Is created for each boundary point, the attribute of the boundary point along the dividing line in a certain direction is obtained based on the connection relationship of the label value of each area, and the attribute is described in a list in a certain direction. Based on the connection relationship, the attribute of the boundary point along the boundary of the divided area in the direction orthogonal to the fixed direction is obtained, the attribute is described in a list in a certain direction, and in a certain direction based on the attribute described in each list. Squirrel in orthogonal direction Seeking a connection length of the one in which the length is determined that corresponds to the boundary of the target region sequence of points of the boundary points corresponding to the list of predetermined length or more, to identify the boundary of the target area.

[Action]

By the attributes described in the list, the connection relationship of the boundary points of each area is expressed in a structured structure,
From this structure, the boundaries of the region of interest are identified.

〔Example〕

A case where the target area identification method in the color image processing apparatus according to one embodiment of the present invention is applied to traveling control of an autonomous traveling vehicle will be described below. According to the method of the present embodiment,
The traveling course of the traveling vehicle is automatically recognized, and the traveling vehicle determines the steering angle of the steering, the amount of fuel injected to the engine, and the like based on the recognized traveling course, and travels autonomously.

FIG. 45 is a block diagram showing a schematic configuration of the entire color image processing apparatus according to the present embodiment.

The color image processing apparatus includes a color camera 101 that captures road information and an ISH conversion unit that performs ISH conversion of the captured RGB information.
102, a color processing unit 103 that extracts a road candidate region or the like from the ISH-converted image information, a labeling hardware unit 104 that performs a labeling process on the color-processed image,
It is broadly divided into a CPU processing unit 105 that executes a merging process or the like on the labeled image area. The ISH conversion unit 102 includes a color camera input board, an ISH conversion board, a filter, and the like.

FIG. 2 is a schematic flowchart showing an image processing method in the color image processing apparatus.

The road image 106 is fetched as RGB information by the color camera 101, and is converted into brightness (I), saturation (S), and hue (H) images in the ISH conversion unit 102 (step 20).
1). On the basis of these images, the color processing unit 103 extracts a road candidate area serving as a basis of the traveling course (step 202). Here, it is determined whether or not the low-luminance threshold value flag in the status register of the CPU is turned on (step 203). This flag indicates whether or not the captured original image has a low-luminance area such as a shadow or a discolored portion.
In 3, a low-luminance area is extracted (step 204). The extracted road candidate area and low-luminance area are labeled in the labeling hardware unit 104, and the connection relationship between the areas is determined. On the basis of this determination result, when the respective areas are in a relationship to be merged, the CPU processing unit 105 executes a merge (merge) process (step 205).

Next, it is determined whether or not the high luminance threshold value presence flag in the status register of the CPU is turned on (step 206). If the low luminance threshold existence flag is not turned on in step 203, this step
Step 206 is executed. The high-luminance threshold value presence flag is a flag indicating whether or not the captured original image has a high-luminance area such as a sunlit area or a discolored part. A luminance area is extracted (step 207). The extracted road candidate area and high-luminance area are labeled in the labeling hardware unit 104, and the connection relationship between the areas is determined. On the basis of this determination result, when the respective areas are in a relationship to be merged, the CPU processing unit 105 executes a merge (merge) process (step 208).

Based on the thus merged road candidate areas, the left and right boundary edges of the area, that is, the road edge boundary lines are obtained as a point sequence. The running course based on the original image captured this time is recognized based on the point sequence information (step 20).
9). Thereafter, the process returns to step 201, and the same processing as described above is repeatedly executed for the image information subsequently obtained along with the movement of the autonomous traveling vehicle, and the traveling control of the autonomous traveling vehicle is executed.

Next, the above processing content will be described in more detail based on the configuration diagram of the color processing apparatus shown in FIG.

The color camera 101 is fixedly installed on the body of the autonomous traveling vehicle, and the color camera 101 captures a road image 106 located in front of the traveling vehicle as RGB information. The RGB information is given to the conversion processing unit 107 of the ISH conversion unit 102. In this conversion processing unit 107, “ISH conversion processing of a color image using a ROM table” which will be described in detail later is executed, and the RGB road image information is converted into a brightness (I) image 108, a saturation (S) image 109, and a hue. (H) It is converted into each image information of the image 110.

The brightness image 108 is given to the road candidate area extracting means 111,
The road candidate area image 113 based on the brightness image is extracted by a “running course extraction method using repeated threshold processing using a template image” described in detail later. The saturation image 109 is given to the road candidate area extracting means 112, and the road candidate area 114 based on the saturation image is processed in the same manner as described above.
Is extracted. The threshold value for the area division in this method is determined by “threshold setting means based on the shape of the feature amount histogram in the repeated threshold processing” described later. Obtained road candidate area images 113 and 114
Is given to the logical product operation means 115, the common part of each road candidate area obtained from the brightness and the saturation is extracted, and a new road candidate area image 116 based on the color information is obtained.

When a low-luminance area is connected to the road candidate area extracted from the brightness image 108, the low-luminance threshold value flag in the status register 117 of the CPU is turned on. When this flag is on, the low-luminance area extraction means 118 takes in the brightness image 108. Then, a low-brightness area is extracted by “extraction of a shadow or a high-brightness portion from a traveling course focusing on a difference in brightness” which will be described in detail later. The extracted low luminance area becomes a low luminance image 119. The low-luminance area image 119 is logically ANDed with the road candidate area image 114 extracted from the saturation image 109 by the AND operation means 120, and the low-luminance area has a saturation similar to that of the road candidate area. The region is extracted. The extracted image information of the low luminance region is added to the road candidate region image 116.

When a high-luminance area exists in the brightness image 108, a high-luminance threshold value flag in the status register 117 of the color processing unit is turned on. When this flag is on, the high-luminance area extraction means 121
Capture 08. Then, “extraction means for shadows and high-luminance parts from the traveling course focusing on the difference in brightness” described in detail later
, A high-luminance area is extracted. The extracted high luminance area becomes the high luminance area image 122. This high brightness area image 122
In the logical product calculating means 123, a logical product is taken with the road candidate region image 114 extracted from the chroma image 109, and a region having a similar chroma to the road candidate region is extracted from the high luminance regions. The extracted image information of the high luminance area is added to the road candidate area image 116.

By the above means, the road candidate area image 116 has a road candidate area image 113 having a pixel value of 1 and a low luminance area image having an image value of 2
119 and a high-luminance area image 122 having a pixel value of 3 are included.

The road candidate area image 116 is provided to the labeling processing unit 124. The labeling processing unit 124 labels each area of the given image to create a label image 126. Also,
At the same time, the labeling processing unit 124 calculates the area and the center of gravity of the same label area. Each of these operation values becomes an operation value 125 corresponding to the labeling processing unit 124. This labeling processing is executed by a “labeling processing device” described in detail later.

The CPU fetches the label image 126 into the small area removing unit 127. Here, it is assumed that a small region caused by noise or the like or a region whose center of gravity is located above the horizon does not correspond to a road region, and these regions are removed from each label image. The label image from which the small area has been removed from the road candidate area by the small area removing means 127 becomes a new label image 128. Also, of the calculation values 125 (feature values) for each label attached to each region, the feature value of each region that has not been removed by the small region removal means 127 is described in List 1.

Based on the feature amount 129, the label image 128, and the road area 116 stored in the list 1, each label region of the road candidate region image, each label image of the low luminance region image, and each label region of the road candidate region image and the high When the respective label images of the luminance area images are in a relation to be merged, the area merging means 130 executes the merging process, and the new label image 131 is stored in the memory 1. The above merging process is executed by "merging means for a plurality of areas" described later in detail.

Road position coordinates 132 at the left and right ends of the road area are calculated based on the finally obtained label image 131. Based on the road end position coordinates 132, point sequence data 133 corresponding to the road end
Is found. The process of obtaining the road edge is executed by "means for obtaining a travelable range" described later in detail, and by "an internal representation method of a running course of various shapes" which is a feature of the present invention. The point sequence data 133 is sent to a data management unit that performs overall management of image processing data for traveling control of the autonomous traveling vehicle, and the color image processing ends.

Next, regarding the “ISH conversion processing of a color image using a ROM table” for converting RGB data obtained by imaging with a color camera into respective data of lightness I, saturation S, and hue H,
This will be described below with reference to FIG.

First, an original image 301 is input as RGB data from a color camera. Since the RGB image data is mixed in the original image 301, each component of the RGB data is separated. Then, each of the separated RGB data is separately stored in each of the three image memories of the R image memory 302, the G image memory 303, and the B image memory 304. Each of these R, G, B image memories 302 to 304 is composed of a plurality of pixel values having 8-bit gradation, and is subjected to ISH conversion as follows.

First, when reading each of the 8-bit RGB data, only the upper 6 bits are read, and the lower 2 bits are discarded.
That is, by taking the upper 6 bits, the pixel value of 8 gradations is approximated to the pixel value of 6 gradations. The numerical value of the upper 6 bits varies between 00 and 3F (hex) in hexadecimal (changes between 00 and FC (hex) considering the truncated lower 2 bits). The value of the upper 6 bits is 40 (hex).
The R, G, and B values are the values divided by. Each of these values of R, G, and B changes in the range from a real number of 0 to about 1.

Each value of R, G, B divided by (R + G + B)
Assuming r, g, b, the conversion from the R, G, B values to the I, S, H values is performed according to the following equation. Here, min (r, g, b) indicates the value of the feature value having the minimum value among the feature values of r, g, and b.

I = (R + G + B) / 3 (1) S = 1− (1/3) · min (r, g, b) H = 1/2 + (1 / π) · arctan ｛(3) ^1/2 · ( gb) / (2r-gb)｝ This conversion is performed for each pixel of R, G, and B in raster scan order, and the converted I, S, and H values are all values from 0 to 1. It is a real number. The ROM 305 stores conversion table data from RGB to lightness I, the ROM 306 stores conversion table data from RGB to saturation S, and the ROM 307 stores conversion table data from RGB to hue H. Each of these ROMs 305 to 307 functions as a look-up table. In addition, each ROM 305-30
The address at which the data is stored in 7 is determined by the upper 6 bits of the 8-bit numerical values of the R, G, B values before conversion. The storage capacity of each of the ROMs 305 to 307 is 18 bits (= 256 Kbytes).

Each feature amount stored in the ROMs 305 and 306 is immediately read out as brightness data 308 and saturation data 309 in response to a fetch instruction from the CPU, and is supplied in real time to required image processing. Further, the feature amount of the hue data 310 read from the ROM 307 is further averaged by removing noise by a 3 × 3 average value filter 311. Therefore, the hue data 312 provided for each image processing is smoothed and noise-free data. Each of the read feature amount data (brightness 308, saturation 309, hue 310 and 312) is 8-bit approximate data in which the lower 2 bits are 0 and the upper 6 bits are valid.

Next, an example of the IHS conversion processing using the present algorithm will be described with reference to FIGS. 4 to 7 while comparing with an IHS conversion processing example not using the present algorithm.

(A) of each figure is an outline of an original image captured by a color camera. That is, FIG. 4 (a) shows a scene in which the traveling path curves far away, and a guardrail is installed on one side of the traveling path, and trees are overgrown in the far side of the guardrail. FIG. 5 (a) shows a traveling road where the traveling road edge is partitioned by weeds or the like, and the scene includes a house, trees, and the like in the distance. FIG. 6 (a) shows a road on a highway at night, where the road surface is a scene lit by moonlight and a small amount of light by a lamp. FIG. 7 (a) shows a daytime running path in good weather, and the roadway is shaded by trees in a block wall.

4B to 7B, (b-1) and (b-2) are histograms using the lightness I as a feature amount, and (c-1) and (c-2) in each of FIGS. The histogram in which the degree S is a feature amount, (d-1) and (d-2) of each figure are 3
A histogram using the hue H ′ before being applied to the × 3 average value filter 311 as a feature amount, (e-1) and (e−
2) is a histogram having the hue H after being subjected to the 3 × 3 average value filter 311 as a feature amount.

The vertical axis of each histogram indicates the number of pixels of each feature amount, with 1/4 of the entire screen being the maximum. The horizontal axis of each histogram indicates the degree of each feature amount of lightness I, saturation S, and hue H ′, H. The degree of each feature amount is displayed so as to increase as the distance from the origin increases, and the degree of each feature amount assigned to each numerical value of 0 to FF (hex) is divided into 64 and displayed. The number of pixels of the original image is slightly smaller than 512 × 512. This is because the R, G, and B data all have a portion of 0 in the peripheral portion of the image, and each histogram is represented by a data value including this portion.

Also, (b-1), (c-1), (d-1),
(E-1) is a histogram obtained based on the conventional method, which is obtained by the CPU directly accessing each image memory and converting the R, G, B data into I, S, H data according to a conversion formula. It is a thing. In contrast, (b-
2), (c-2), (d-2), and (e-2) are histograms obtained based on the algorithm according to the method of the present embodiment. S, H
This is obtained by storing a conversion table for each feature amount and reading out the conversion table.

As shown in each of (b-1) and (b-2) and each of (c-1) and (c-2) in FIGS. 4 to 7, the brightness I and the saturation S It can be seen that there is no significant difference between the histogram distribution according to the conventional method and the histogram distribution according to the conventional method. This indicates that the function of converting RGB data to ISH data by the present method has no inferior aspect to the conversion function by the conventional method. On the other hand, (d) of each figure
-1), 3 × 3 average value filter 31 shown in (d-2)
The overall distribution of the histogram distribution of the hue H ′, which is the feature value before the multiplication by 1, is different between the conventional method and the present method. This is because (gb) and (2r-) in the expression for converting the RGB data to the hue H in the image used in this example.
Since the value of g-b) takes only a very limited value near 0, the information is extremely discretized by the compression of the data into 6 bits by the present method.

However, the histogram distribution obtained by applying the hue H ′ to the 3 × 3 average value filter 311 is not enough for the distribution according to the conventional method as shown in (e-1) and (e-2) of each figure. It is corresponding. The calculated value of the hue data is an unstable value that easily changes due to small noise in the RGB data, and the hue image has very large noise. Therefore, it is appropriate to perform some kind of smoothing on the calculation value result of the hue conversion as in the present method, and by performing this smoothing, the feature amount is compressed to 6 bits and processed. It is understood that no problem occurs. Since the calculated value of the hue is an unstable value that easily changes due to noise, the same distribution as the histogram distribution by the conventional method was obtained by passing the data of the hue H ′ through an average filter.

As described above, by performing the conversion process on each of the I, S, and H feature amounts in advance by using the ROMs 305 to 307 as a look-up table, the CPU is stored in each image memory every time processing is required as in the related art. There is no need to access and perform calculations. As a result, the arithmetic processing speed at the time of data conversion according to the present embodiment can be executed at a high video rate, and the processing speed is improved. Also, from the R, G, B data described above, I, S, H
Even if the conversion formula for data changes, it can be dealt with by the same hardware. In other words, the effect of the change of the conversion formula is only the change of the storage contents of the ROMs 305 to 307. Therefore, the hardware is not changed by changing the conversion formula. Also, since each pixel value of R, G, B is compressed to 6 bits, the amount of hardware can be reduced. Further, when the hue H is read from the look-up table, the data is passed through the 3 × 3 average value filter 311, so that the adverse effect of compressing the data to 6 bits, for example, the discretization of the histogram can be prevented.

Next, “threshold setting means based on the shape of the feature amount histogram in the repetitive threshold processing” will be described. This means is performed as a pre-process of the color image processing for extracting the travel road area.

FIG. 8 is a flowchart showing the outline of this processing, and shows the processing when the contrast of the original image data obtained from the color camera installed in the traveling vehicle is low.

First, a histogram representing a frequency with respect to a color feature (brightness or saturation) is created based on RGB digital image data of an original image obtained from a camera. The horizontal axis of this histogram is set to the color feature (lightness or saturation), and the vertical axis is set to the frequency of the feature. Since the contrast of the original image is low, the histogram is distributed toward the origin of the histogram. On the other hand, an image having a low contrast tends to have a single-peak mode, and a clear valley does not occur. For this reason, in general, a histogram is created after performing an image enhancement by performing a predetermined operation on each pixel value. In the case of this method, however, the histogram is expressed in the horizontal axis direction on the histogram data. Image enhancement is performed by simply stretching (step 801).

Then, the left end processing (step 802) and the right end processing (step 803) of the emphasized histogram are executed.
Next, from the frequency distribution state of the histogram, a temporary peak (peak) having a high frequency for the feature value and a temporary valley having a low frequency for the feature value are set on the peak / valley table (step 804). The valleys are evaluated on this table based on the obtained temporary valley frequencies (step 805).
Further, the valley is evaluated again on the table based on the peak adjacent to the valley (step 806). Finally, the evaluated peak / valley table is smoothed, and a valley effective for the division between the target region of the region division and the background region is extracted from the relative relationship between the peak and the valley (step 807).

On the other hand, in parallel with the above processing, the Otsu's discriminant analysis method is applied to the histogram created in step 801 to obtain a region segmentation threshold value by this discriminant analysis method. And
The obtained threshold value is compared with the position of the valley extracted in step 807, and the frequency of the valley located near the threshold value is set as a true threshold value. to correct.

 Next, details of each process will be described below.

The setting of the temporary peak and the temporary valley in step 804 is performed as follows. That is, provisional peaks and valleys are determined based on the frequency distribution state for the feature amount. Specifically, the states of the peaks and valleys are shown in FIG.
The state is divided into six states shown in FIG. Here, the point of interest on the histogram is hi (the subscript i means an arbitrary point when N points are evenly allocated along the horizontal axis of the histogram, and has a value of 0 to N-1). ), Attention point hi
An adjacent point having a smaller feature amount is defined as hi-1, and an adjacent point having a larger feature amount than the target point hi is defined as hi + 1. Further, the difference between the frequencies of the point of interest hi and the adjacent point hi-1 is represented by the pitch pi1 (pi1 = hi-hi-
1) pitch difference between each frequency of adjacent point hi + 1 and point of interest hi
pi2 (pi2 = hi + 1-hi).

In the range of i = 1 to N−2 (excluding both end points of the histogram), the peak / valley table value pkti is set as follows. That is, when the sign of pi1 is different from the sign of pi2, (1) If pi1 ≧ 0 and pi2 ≦ 0, the table value pk
ti = 1 (2) If pi1 ≦ 0 and pi2 ≧ 0, the table value pk
ti = -1 FIG. 9 (a) shows a state where pi1> 0 and pi2 = 0, and therefore the table value pkti = 1. FIG.
pi1> 0 and pi2 <0, so that the table value pkti = 1. FIG. 3C shows that pi1 = 0 and pi2
<0, therefore the table value pkti = 1.
FIG. 6D shows a state where pi1 <0 and pi2 = 0.
Therefore, the table value pkti = -1. (E) of FIG.
1 <0 and pi2> 0, therefore the table value
pkti = -1. FIG. 7F shows that pi1 = 0 and pi2
> 0 and therefore the table value pkti = -1.

In this way, from the relation of the relative frequencies of the adjacent points, the states of FIGS. 7A, 7B, and 7C are calculated to be pkti = 1, and the point of interest hi is a temporary peak Is determined.
In the states shown in FIGS. 4D, 4E and 4F, it is calculated that pkti = −1, and the point of interest hi is determined to be a temporary valley.

In addition, the valley evaluation processing based on the frequency in step 805 is performed as follows. That is, when the point of interest hi is a temporary peak at i = 1 to N-2 (pkti = 1), (1) the ratio between the adjacent point hj and the point of interest hi is smaller than 0.1 (h
j / hi <0.1) If the adjacent point hj is on the left side of the point of interest hi,
The table value pktj corresponding to the adjacent point hj is set to −1 (pktj
= -1).

(2) The ratio between the adjacent point hm and the point of interest hi is smaller than 0.1 (h
m / hi <0.1) If the adjacent point hm is to the right of the point of interest hi,
Set the table value pktm corresponding to the adjacent point hm to -1 (pktm
= -1).

The valley evaluation processing based on the adjacent peak in step 806 is performed as follows.

When i = 0 to N−1, (1) When the point of interest hi is a temporary valley (pkti = −1), a temporary peak on the left side of the valley (neighboring point hk, table value pktk =
Let 1) be called topL.

A temporary peak on the right side of the valley (neighboring point hj, table value pktj =
Let 1) be called topR.

(2) When topL and topR are both present at the point of interest hi, the ratio l (l = hi / topL) between the point of interest hi and topL is the point of interest hi
If it is smaller than the ratio r between r and topR (r = hi / topR), (l <
r), add 1 to the table value pktj.

If the ratio l (hi / topL) is larger than the ratio r (hi / topR) (l ≧ r), 1 is added to the table value pktk.

(3) If both topL and topR are present at the point of interest hi and the ratio l <0.5 or the ratio r <0.5, 1 is subtracted from the table value pkti corresponding to the point of interest hi.

(4) In cases other than the above (3), the table value pkti
To -4.

(5) When the temporary peak topR exists only on the right side of the valley,
If the ratio r <0.5, 1 is subtracted from the table value pkti.

(6) When the temporary peak topL exists only on the left side of the valley,
If the ratio l <0.5, the table value pktj corresponding to the adjacent point hj
Minus 1 from

Also, in the peak / valley table smoothing process in step 807, that is, in the peak / valley table pkt,
When the distance between the obtained valleys is sufficiently short, the smoothing process is performed as follows.

At i = 1 to N-1, the table value pktj corresponding to the point of interest hi is -2, and the table value pktj of the valley located on the right side of the valley is also -2, and the distance between these valleys ( When ji) is smaller than a predetermined threshold value, (1) If the frequency of the point of interest hi is higher than the frequency of the adjacent point hj (hi> hj), the table value pkti is set to 0 (pkti).
= 0).

(2) If the frequency of the point of interest hi is lower than the frequency of the adjacent point hj (hi ≦ hj), the table value pk of the valley located on the right side
Set tj to 0 (pktj = 0).

Next, a specific example using the above method will be described below.

For example, assume that the feature amount histogram shown in FIG. 10 (a) is obtained. The horizontal axis in the figure is the lightness or chroma color feature amount, and the vertical axis is the frequency of the feature amount in the image. When the processing shown in FIG. 8 is performed on this histogram, the table values of the peak / valley table pkt shown in FIG. 8B are 1 to 3 shown in the leftmost column of FIG. The transition is as follows along the number 8. As a result of this transition, the numerical value shown in the lower row is obtained as the final table value. The positions at which the table values in FIG. 3B are described correspond to the feature amounts A to L of the histogram in FIG. That is, each listed table value corresponds to a feature value above the described position, as indicated by a dotted line.

First, a left end process and a right end process of the feature amount histogram are executed (steps 802 and 803), and then a temporary peak and a temporary valley are set (step 804). This setting is performed in accordance with the processing of step 804 described above, and the table values obtained as a result are as shown in the table of No. 1. Next, valley evaluation based on frequency (step 805) and valley evaluation based on adjacent peaks (step 806) are performed according to the above-described processing.

When the feature value is the valley of A, only topR exists on the right side of the valley, and the ratio r between the frequency in the feature value A and the frequency of the temporary peak on the right side of the valley is 0.5 or less (hi / topR). <0.5).
For this reason, 1 is subtracted from the table value corresponding to the feature amount A, and as a result, the table value becomes -2 and the table changes to the table indicated by the number 2.

When the feature amount is a valley of D, to
pL and topR are both present. Moreover, the ratio 1 between the frequency of this valley and the frequency of the temporary peak on the left is 0.5 or less (hi /
topL <0.5). Therefore, the table value p corresponding to the feature amount D
One is subtracted from kti, resulting in a table value of -2. The ratio r and the ratio r between the frequency of each temporary peak value and the frequency of the valley on both sides of the valley having the feature value D are larger in the ratio r (l <r). Therefore, by adding 1 to the table value pktj of the temporary peak (feature amount E) located on the right side of the valley, the table value becomes 2. As a result, the peak / valley table becomes the table indicated by No. 3.

In addition, both topL and topR are on both sides of the valley where the feature value is F.
And one of the ratios l or r is less than 0.5 (l, r
<0.5). Therefore, the table value pkti corresponding to the feature value F
Is subtracted from 1 to make the table value −2. Further, the ratio 1 is larger than the ratio r (l ≧ r). Therefore, 1 is added to the table value pktk of the temporary peak (feature amount E) located on the left side of the valley, and the table value is set to 3. As a result, the table changes to the table indicated by No. 4.

Also, on both sides of the valley where the feature value is H, both topL and topR
And one of the ratios l or r is greater than 0.5 (l, r
≧ 0.5). Therefore, the table value pkt1 corresponding to the feature amount H
To -4. Further, the ratio 1 is smaller than the ratio r (l <r). Therefore, 1 is added to the table value pktj corresponding to the temporary peak (feature amount I) on the right side of the valley. As a result, the table value corresponding to the feature value I becomes 2, and the table changes to the table indicated by No. 5.

Further, both sides of the valley of the feature amount J have topL and topR, and the ratio r is smaller than 0.5 (r <0.5). Therefore, 1 is subtracted from the table value pkti corresponding to this valley. Also,
The ratio l is greater than the ratio r (l ≧ r). Therefore, the table value corresponding to the temporary peak (feature amount I) located on the left side of the valley
Add 1 to pktk and set the table value to 3. As a result,
The table changes to the table indicated by number 6.

Also, the valley of the feature amount L has only topL on the left side of the valley,
The ratio 1 is less than 0.5 (l <0.5). Therefore, 1 is subtracted from the table value pkti corresponding to this valley. As a result, the table value becomes -2, and the table shifts to the table indicated by number 7.

Next, data is smoothed as described above for the peak / valley table pkt of number 7 obtained in this manner (step 807). That is, the distance between the valley hi of the characteristic amount D and the valley hj of the characteristic amount F located on the right side thereof is smaller than the predetermined threshold value, and the table value of each valley is -2. Further, the frequency of the valley of the feature F is higher than the frequency of the valley of the feature D (hi ≦ hj). Therefore, the table value pktj corresponding to the valley of the feature value F is set to 0. As a result, the table is eventually
10 The table shown at the bottom of FIG.

In this final table, each point on the histogram corresponding to the feature values A, D, H, J, L is obtained as a valley, but the valleys A, L at both ends of the histogram are not targeted, and ,
The valley having the table value of -4 (the feature amount H) is not a target. That is, the valleys effective for dividing the data image into the target area and the background area are valleys corresponding to the feature values D and J in which the table value-2 is surrounded by double circles. Of these valleys, the frequency of valleys close to the region division threshold obtained by applying the Otsu discriminant analysis method to the histogram of FIG. 10 (a) becomes the true region division threshold. By correcting the ambiguous result of the Otsu's discriminant analysis method in this way, it is possible to perform a highly accurate traveling road discrimination with a small error.

Next, a description will be given of a “running course extraction method using repeated threshold processing using a template image”. In the following description, it is assumed that the autonomous traveling vehicle travels on a course sectioned by grass, soil, and the like, and a case in which a traveling road region is extracted will be described.

FIG. 11 is a flowchart showing an outline of an algorithm for recognition of a traveling road.

First, with the color CCD camera mounted on the traveling car,
The scene of the traveling road is imaged (step 1101). Then, the captured RGB color image signal is input to a color camera input device, and the input RGB original image data is converted into ISH data as described above (step 1102). Based on the feature image converted to the ISH data, a histogram representing the characteristics of each feature and the number of pixels is created by the color image processing device as described above (step 1103).

Next, based on the created histogram, the threshold value processing is repeatedly applied to extract the traveling road as a binarized image. Since the image extracted in this manner has noise and finely divided areas, the following processing is executed. That is,
Each area is labeled by a labeling device described later (step 1104). Then, the area and the center of gravity of each labeled area are measured, and the area having the center of gravity higher than the horizon position calculated from the camera mounting position and the area having a smaller area are removed (step 1105).

Next, based on the traveling road image first extracted in the color image processing device, an area having a brightness lower than that of the traveling road area and a bright area are obtained, these areas are merged, and a relationship between the respective areas is determined. This is described (step 1106).
A travelable range, that is, a road edge is detected based on the description of the area merging and the area relationship, and an image capture time is added to the travelable range information. Then, the travelable range information is transmitted to the data management unit included in the image processing apparatus (step 1107). Thereafter, the process returns to step 1102 to repeat the above processing.

FIG. 12 is a block diagram showing details of a process for extracting a travel road region.

First, lightness data I and saturation data S are obtained from RGB data of a captured original image, and each data is stored as a feature image in an image memory (block 1201). Next, based on the brightness data I stored in the image memory, a histogram having brightness as a feature amount is created (block 1202).
The number of divisions of the feature amount on the horizontal axis of this histogram is 40 points, and the number of points is large enough to globally extract the traveling road. Next, the generated histogram is normalized (block 1203). Then, a well-known Otsu's discriminant analysis method is applied to the normalized histogram to calculate a threshold value for distinguishing between the roadway region and the background region (block 1204).

On the other hand, a peak having a large number of feature amount pixels and a valley having a small number of feature amount pixels are obtained based on the shape of the created histogram as described above, and a peak / valley table which is a list of peaks and valleys is created. (Block 1205). The value of each feature amount at each peak and valley is set as a candidate value for threshold processing.

Since the threshold value obtained by the Otsu discriminant analysis method obtained in block 1204 may include an error, the threshold value obtained by the Otsu discriminant analysis method Is corrected. That is, block 12
The threshold value obtained in step 04 is compared with the candidate value obtained in block 1205.
The candidate value of 05 is set as a threshold value for distinguishing the travel road area from the background area (block 1206). Next, the brightness image is binarized using the threshold value (block 1207).

The brightness image is composed of 512 × 512 pixels and is represented as I (i, j). Here, i, j is an integer satisfying the conditional expression 0 ≦ i, j ≦ 511. Also block 12
It is assumed that the threshold value obtained in 06 is X1, the minimum feature value of the feature amount range to be divided into regions is a threshold value X0, and the maximum feature value is a threshold value X2. The initial values of the thresholds X0 and X2 are 0 and FF (hex). If the brightness image I (i, j) satisfies the following conditional expression, the memory M
Write FF (hex) to 1 (i, j) (block 1208).

X0 ≦ I (i, j) <X1 If the brightness image I (i, j) satisfies the following conditional expression, FF (hex) is written to the memory M2 (i, j) (block 1209). .

X1 ≦ I (i, j) <X2 Next, the template image stored in the ROM is read (block 1210). A template area is set in this template image so that the most stable travel path information can be obtained. This area setting is performed based on the depression angle, the angle of view, and the focal length of the color camera attached to the autonomous vehicle, and the memory corresponding to the template area has FF (he
x) is stored. Next, the degree of overlap between the read template image and the memories M1 and M2 is calculated (block 1211). That is, the template image and the memory
The logical product of the memory M1 and the memory M2 is calculated, and the number of pixels whose logical product is “1” is accumulated for each of the memories M1 and M2.

Next, a ratio of the number of pixels accumulated in each memory to the number of pixels in the template area is calculated. It is assumed that the memory information in which the ratio is 50% or more includes the travel road area. Further, the image stored in the memory M1 or M2 in which the ratio is 50% or more is repeatedly divided into regions as follows. I do. Further, by selecting a memory based on this ratio, the image is divided into regions by the threshold value X1, and the division is also performed on the histogram.

In other words, the area stored in the memory M1 whose characteristic amount is in the range of X0 to X1, and the memory whose characteristic amount is in the range of X1 to X2
This means that the area is divided into the area stored in M2. Also,
When the degree of overlap between each of the memories M1 and M2 and the template image does not exceed 50%, the memory M3 storing the sum of the memories M1 and M2 is selected (block 1212).
Area division based on the memory M3 is performed next.

For example, when the degree of overlap between the memory M1 and the template image is high, an area constituted by the feature values within the threshold values X0 to X1 is set as a road candidate area (block 121).
3). Then, the road candidate area is further divided into areas. Threshold X for this repetition area segmentation
1 ′ is a candidate value of the valley obtained in the block 1205 and is within the range of the threshold value X0 to X1 (block 121).
Four). If there is no valley candidate value within this range, the repetitive division processing is not performed.

Minimum value X of the feature amount range to be iteratively divided into regions
0 ′ becomes the threshold value X0 which is the same as the previous area division,
The maximum value X2 'becomes the threshold value X1. The area constituted by the feature amounts within this range is re-binarized in block 1207, and then the same processing as the previous area division is performed, thereby repeatedly performing area division. This repetition is performed until there are no more valley candidate values obtained in block 1205 within the range of the feature amount of the road candidate area to be divided. By repeatedly executing the processing in this manner, the first road image area is finally obtained.

If there is a shadow area darker than the brightness of the obtained road image area and a bright high-luminance area, thresholds for obtaining these areas are set as described later (block 1215). Then, based on the threshold value, each region is divided into regions in the same manner as described above, and a shadow region and a high luminance region are obtained. When the saturation of the obtained shadow area and the saturation of the road image area are similar according to the processing result based on the saturation image performed at the same time, the logical product of each of them is calculated, and the final is determined as one area. Low brightness area. At the same time, when the obtained saturation of the high luminance area is similar to the saturation of the road image area, the logical product of each of them is taken and one area is set as a final high luminance area. Further, the connection relationship between the road image region and the low-luminance region is checked, and if there is a relationship that can be combined, a merging process is performed. Similarly, the connection relationship between the road image area and the high brightness area is checked.

When there is such a relationship that merging is possible, merging processing is performed. As a result, when there is a shadow on the traveling road or when the position of the vehicle itself is in the shadow and direct sunlight is radiated to the far side of the traveling road and there is a high luminance portion, by performing this merging process, the actual traveling road is A travel path region having a suitable shape can be obtained.

Although the above processing is for a lightness image, the same processing is performed for a saturation image. However, block 1215
This process is not performed because it is unique to the brightness image. In the processing result by this saturation image, when the R, G, and B of each area are similar colors having similar values,
The histogram distribution of each area has a similar shape. This is based on the above-described conversion formula (S
= 1−min (r, g, b) / 3).

Therefore, it is difficult to distinguish each area based on the saturation in the sky on a cloudy day or a general pavement road where the values of R, G, and B are similar. However, when there is a color difference and the R, G, and B values of each area are not similar, each area can be distinguished even if there is no difference in brightness of each area. Therefore, in order to accurately extract the road image area under any scene, two feature amounts of lightness and saturation are used. Then, by taking the logical product of the two types of road images extracted from the brightness image and the saturation image, it is possible to obtain highly-accurate traveling road area information.

FIG. 13 shows a process of extracting a road candidate region in a road image region extraction process.

First, a feature amount histogram is obtained from the brightness feature amount image 1301.
Create 1302. The horizontal axis of the histogram 1302 indicates lightness, and the lightness is represented by numerical values of 0 to FF (hex). The vertical axis indicates the number of pixels in the original image at each brightness. Otsu's discriminant analysis method is applied to the histogram 1302, and a threshold value c for distinguishing a road region from a background region is obtained. Further, based on the shape of the histogram 1302, a peak and a valley of the histogram are obtained, and a peak / valley table is created. Then, the obtained valley is set as a candidate value of the threshold value for the repeated threshold value process. In this histogram 1302, the feature values a, b, d, e, and f are candidate values.

The threshold value c obtained by the Otsu discriminant analysis method is corrected by the candidate value obtained from the peak / valley table. That is, the candidate value b closest to the threshold value c is set as the corrected threshold value. Then, the feature amount image is determined by the threshold value b.
Binarize 1301. As a result, the feature amount is 00 (hex) to b
Divided image 1303 and divided image 1 whose feature amount is b to FF (hex)
304 is obtained. At this point, each divided image 1303,
It is not known which of the images 1304 includes the travel path area.

Therefore, the template image 1305 assuming the road position
And the logical product of each image 1303 and 1304 is calculated, and the degree of overlap with each image is calculated. Below the template image 1305, a trapezoidal template area is set as shown. As described above, FF (hex) is stored in the memory element corresponding to the template area, and 00 (hex) is stored in the memory element corresponding to the background area of the template area. In the case of this example, the image 1304 has the most overlapping portion with the template image 1305. Therefore, according to the calculation result of the degree of overlap, the feature amount is b to FF (he
It is determined that the traveling road area is included in the image 1304 of x), and the road candidate area image 1306 corresponding to the image 1304 is obtained.

Since other candidate values still remain between the feature values of b to FF (hex), the road candidate area image 1306 is further added.
Is divided into regions. That is, the road candidate area image 13
06 is binarized by a threshold value d. By this binarization, a divided image 1307 whose feature amount is b to d and a feature amount of d to FF (hex)
Is obtained. Next, each of the obtained images 13
07, 1308, template image 1
Calculate the degree of overlap with 305. In this case, image 1308
Has a higher degree of overlap with the template image 1305, it is determined that the image 1308 having the feature amount of d to FF (hex) includes the travel road area, and the road candidate area image 1309 corresponding to the image 1308 Is obtained.

Since another candidate value e still remains between the feature amounts of d to FF (hex), the road candidate region image 1309 is binarized by the threshold value e. As a result of this binarization, a divided image 1310 having a feature amount of de to e and a divided image 131 having a feature amount of e to FF (hex) are obtained.
1 is obtained. Then, each image 1310,
The degree of overlap between 1311 and the template image 1305 is calculated. In the case of this example, since the image 1310 has a higher degree of overlap with the template image 1305, it is determined that the image 1310 having the feature amounts d to e includes the travel path region, and the image 1310 corresponds to the image 1310. The obtained road candidate area 1312 is obtained.

Since no other candidate values remain between the feature amounts d to e, this road candidate area 1312 is the final road area 2
Value image. The above processing is performed on the brightness image, but the same processing is performed on the saturation image. Thereafter, a logical product of the traveling road region extracted from the brightness image and the traveling road region extracted from the saturation image is obtained to obtain a final traveling road region.

It is general to binarize an image using a threshold value and divide the image into regions. However, as in this embodiment,
By calculating the degree of overlap with the divided image using the template image considering the position of the traveling road, the process of determining the threshold for the next region division performed on the histogram can be performed quickly and simply. . As a result, a travel path area that matches the actual travel path can be obtained at high speed, easily, and at low cost.

Further, the brightness of a scene captured by a camera may change as follows. For example, there is a case where the brightness changes because the traveling direction of the own vehicle is changed by a curve, and a case where the brightness changes because the weather is sunny or cloudy. In such a case, since the feature amount histogram does not always show a fixed shape, a correct travel road region cannot be obtained by region segmentation using a fixed threshold value. However, in this method, it is possible to always accurately extract the traveling road area even if the state of the input image, that is, the brightness changes every moment. This is because, in the present method, a feature amount distribution having the highest degree of overlap with the template image is found, and an optimum threshold value according to the feature amount at that time is set for an image that changes every moment.

2. Description of the Related Art Conventionally, in image processing for region division, an original image is divided into a plurality of regions, and identification processing is performed on the divided images to verify the validity of a road. However, in the above-described method, region extraction is performed by threshold processing in order to extract a target object (road).
As described above, in the conventional method, the target is verified from the processing result, but in the present method, the processing approach is reversed in that the target is targeted from the beginning of the processing. For this reason, it is possible to efficiently extract the road area.

Next, a method of extracting a shadow or a high-luminance portion from a traveling course focusing on a difference in brightness will be described. This method has already been described briefly in the description of the “running course extraction method by repeated threshold processing using a template image”, and this method will be described in detail below.

Regarding the brightness image, the above-mentioned “running course extraction method by repeated threshold processing using template images”,
By applying the "threshold setting method based on the shape of the feature amount histogram in the repetitive threshold value process", a histogram in which the feature amount of the traveling course is distributed was obtained. In this method, an area darker than a road area and an area lighter than a road area are obtained based on the histogram. In addition, this method is processed in the color image processing apparatus.

FIG. 14 shows an outline of processing when the present method is applied to each case where various input images are captured. Case 1 is a case where only a traveling course having a uniform road surface condition is captured as an input image. In this case, the vehicle is located just before the curve of the road that curves to the right. The road candidate area extracted in this case 1 has the same shape as the input image. This is because the input image has only a uniform traveling course, and therefore, the low luminance region and the high luminance region are not extracted by this method.

Case 2 is a case where an input image in which a shadow is partially formed on the road surface of the traveling course and a high-luminance portion is formed far away from the traveling course due to reflected light or the like. In the example in this case, the road curves rightward and has a fork that curves leftward just before the right curve. Your vehicle is located before these curves. The road candidate area extracted in this case 2 has a shape excluding a dark area where a shadow is formed and a bright area which is a high luminance area. In addition, a shadow portion formed on the front side of the road can be individually extracted by extracting a low-luminance region by the present method. In addition, a high-luminance area due to reflected light formed far away on a road can be individually extracted by extracting a high-luminance area.

Case 3 is a case where an input image in which a shadow of a tree is formed on the road surface of the traveling course is captured in a case of strong sunshine, and a day when a tree leaks on the road surface. In the example of this case, the own vehicle is located in front of a curve on a road that curves to the left. The road candidate area extracted in this case 3 has the same shape as the shadow formed by the tree leaking day. This is because the shadow due to strong sunlight has a high degree of overlap with the template image. Therefore, a low-luminance region is not extracted by this method. Also,
A distant road area and a part where a tree leaks are extracted as high-luminance areas due to strong sunlight.

Case 4 is a case where a discolored portion is formed in a belt shape along a side band of the road, for example, a case where the road is discolored due to pavement road construction or the like. In the example of this case, the own vehicle is located at a position where the vehicle travels by looking at the discolored portion formed on the road traveling in a straight line to the right. The road candidate area extracted in this case 4 has a shape excluding this discolored portion.
In addition, since the discolored portion has higher brightness than the road region, the discolored portion is extracted as a high luminance region by this method. Further, since there is no shadow or the like on the road surface, a low luminance area is not extracted.

Next, the details of the present method will be described below using the case 2 described above as an example.

FIG. 15 is a histogram created based on the input image captured in Case 2. The feature amount of this histogram is lightness, and this lightness is shown on the horizontal axis. The vertical axis indicates the number of pixels at each brightness.

The feature amount range of the portion A in the drawing is a range including the travel road area, and a range including the pixel corresponding to the travel course most in the above-described “running course extraction method by repeated threshold processing using a template image”. It is the part that is said to be. The range A is a range in which the feature amount is from t1 to t2, and the area is divided using the feature amounts t1 and t2 as thresholds. A portion B located on the left side of the feature amount t1 forms a single crest sandwiched between valleys, and is a feature amount distribution in a dark range where the brightness is lower than the portion A. The feature value range of this B part is t3
To t1, and each of the feature amounts t3 and t1 becomes a threshold for region division. Also, C on the right side of the feature amount t2
The portion is a feature amount distribution in a bright range where the brightness is higher than the portion A, and forms one mountain sandwiched between valleys like the portion B. The feature amount range of the C portion is from t2 to t4, and each of the feature amounts t2 and t4 is a threshold for region division.

The portions B and C shown in the figure form one peak, but such a feature distribution that does not form one peak forms a meaningful region on the image for region division. Not recognized as distribution. Therefore, such a feature amount distribution is not selected as a region extraction target because it is not a superior distribution. In the illustrated example, both the B portion and the C portion have a dominant distribution, but the present method can be applied even if only one of the distributions is dominant.

The feature amount distribution of the portion A corresponds to the road candidate region of case 2 shown in FIG. The feature amount distribution of the portion B corresponds to a shadow region having lower brightness than the road candidate region, and the feature amount distribution of the portion C corresponds to a high brightness portion having higher brightness than the road candidate region. In this method, each region corresponding to each feature amount distribution of the B portion and the C portion adjacent to the A portion is extracted.

First, in order to extract a region corresponding to the B portion of the histogram, the threshold value t3 and the threshold value t1 are used to binarize an input image whose brightness is a feature amount. Further, in the above-described “running course extraction method using repeated threshold processing using a template image”, a road region image is obtained based on an input image having a saturation as a feature amount. The road area image is obtained by first dividing the original image into areas, and is an image in which a dark area such as a shadow or a high luminance area is included in the road area. 1 is stored in the memory element corresponding to the road area, and 0 is stored in the memory elements corresponding to the other background areas. Also,
In the above-described binarized image, 1 is stored in a memory element corresponding to a pixel area having a brightness of t3 to t1, and 0 is stored in a memory element corresponding to another area.
For this reason, by taking the logical product of the road area image and the binarized image, only dark areas such as, for example, shadows on the travel road area are individually extracted.

Further, in order to extract a region corresponding to the portion C, the brightness image is binarized by the threshold values t2 and t4 in the same manner as in the case of obtaining the dark region. Then, by taking the logical product of the road area image extracted from the saturation image and this binarized image in the same manner as in the case of the extraction of the B portion, the high-brightness areas are individually extracted.

Next, a method of obtaining threshold values t3 and t4 required for dividing the B portion and the C portion into regions will be described. The “threshold setting means based on the shape of the feature amount histogram in the repetitive threshold processing” described above
The peak / valley table shown in FIG.
Similarly, a peak / valley table (not shown) is obtained for the histogram shown in FIG. Each table value in this peak / valley table is expressed as pkti, as in FIG. 10 (b). The subscript i is i = 0, 1, 2,... N, N in order corresponding to each table value from the origin side of the graph.
It is assumed to change to -1.

In the histogram of FIG. 15, the table value corresponding to the threshold value t1 is pktj. Then, in the peak / valley table, look at each table value to the left from this table value pkti, and the table value becomes negative up to pkt0
It is determined whether or not pktk (pktk <0) exists. The feature point where the table value becomes negative is a point corresponding to the bottom of the valley. There is a negative pktk by pkt0, and pktk from the feature point of pktj
When the distance to the feature point of tk is smaller than the threshold (pktj
−pktk <threshold value), and a feature amount corresponding to the table value pktk is set as a threshold value t3. Also, pk that becomes negative by pkt0
If there is no tk or the distance exceeds the threshold, it is assumed that there is no dominant area darker than the road area.

The feature amount corresponding to the threshold value t4 on the histogram can be obtained in the same manner as the method for obtaining the feature amount corresponding to the threshold value t3. That is, assuming that the peak / valley table value corresponding to the threshold value t2 is pktm, each table value is looked to the right from this pktm, and pktn which becomes negative by pktN-1
It is determined whether (pktn <0) exists. If there is a negative pktn by pktN−1 and the distance from the pktn feature point to the pktm feature point is smaller than the threshold value (pktn−pk
tm <threshold value), and a feature amount corresponding to this table value pktn is set as a threshold value t4. Also, pk that becomes negative by pktN-1
If there is no tn, or if the distance exceeds the threshold, it is assumed that there is no dominant area brighter than the road area.

As described above, the present method uses the peak / valley table obtained by the “threshold setting means based on the shape of the feature amount histogram in the repetitive threshold value processing” to change the weather and construct pavement roads. For example, even if the road surface condition of the road changes, shadows, high-luminance portions, and discolored portions can be individually extracted.

Next, the “labeling device” will be described in detail below.
The image extracted in the “running course extraction method by repeated threshold processing using a template image” has noise and finely divided regions. Therefore, each area of the extracted image is labeled by the labeling device, and the validity of each labeled area is determined. Based on the result of the labeling process, a region having a center of gravity higher than the horizon position and an unnecessary small region generated by noise or the like are removed.

FIG. 16 is a block diagram showing a schematic configuration of the labeling device, and FIG. 17 is a general flowchart showing an outline of the labeling process. First, the image extracted by the color processing device is loaded into the input memory 1601 via the image bus (NE BUS) (step 1701). This image information is 512 × 512 × 8 bit information, which is used as a multi-valued original image input. Next, primary labeling is performed using a temporary labeling method using a run described later (step 17).
02). This primary labeling is mainly executed in a labeling processor (KLP) 1602, a line buffer memory (LBM) 1603, a label matching memory (LMM) 1604, and the like.

After this primary labeling, the label matching memory LM
Preprocessing is performed to arrange the data arrangement of M1604 (step 1703). After this preprocessing, secondary labeling is performed, and at the same time, feature amounts such as the area and the center of gravity of each region are extracted (step 1704). The processing of steps 1703 and 1704 is mainly executed by the feature extraction processor KLC1605. By this secondary labeling, the label attached to the pixel located at each address is stored in a label memory (LABEL).
M) Store in 1606. At the same time, the area and the center of gravity of each extracted region are stored in the feature memory 1607. After this, LA
The label image information stored in BELM 1606 is output to NE BUS (step 1705).

The number of KLPs 1602 used for label generation is one, and the LBM 1603, which is a line register for run processing, is configured using four label memories KLM described later. Also, L
When the maximum number of temporary labels is 1023, the MM1604 is configured using eight KLMs. KLM when the maximum number of temporary labels is 4095
Are used to configure the LMM1604.

Conventional labeling has been performed only when the input image is a binary image.
By using 2, it is possible to perform labeling on multi-valued images. That is, it is possible to label several types of images at once. For example, as shown in FIG.
It is assumed that 01, 1802, and 1803 are input. These binary images are added to form a 512 × 512 × 8 bit multi-valued image 1804
Is converted to This conversion processing is performed as preprocessing of the labeling processing. Under the control of the labeling processor 1807 (KLP1602) using the label matching memory 1806 (LMM1604) instead of the line register 1805 (LBM1603), the multilevel input image 1804 is labeled. In this labeling, the labeling of each area is organized and finally 512 × 512
Output as a label image 1808 of × 12 bits. By using the labeling processor 1807 (KLP1602), it is possible to perform labeling processing on a multi-valued image, reduce the number of temporary labels using runs, and perform one-scan labeling.

Labeling according to each pixel value is performed on each pixel of the multi-valued input image under the control of the KLP1602. Then, area division is performed based on the connection relationship between the pixels having the same label value, and a new label is generated based on the connection relationship. For example, conventionally, in the case where an area composed of stepped pixels shown in FIG. 19 (a) is formed in an input image, temporary labeling is performed on each pixel in the order of raster scanning, and each pixel is again labeled. The label generation has been performed by scanning for. As a result, at the time of provisional labeling, three types of provisional labels 1 to 3 were required as shown in the figure.

However, according to the labeling method using a run according to the present method, even if there is an input image shown in FIG. 19B composed of step-like pixels similar to FIG. The number of temporary labels can be reduced. That is, as shown in FIG. 3C, each pixel is divided into one row of runs along raster scanning. In the case shown in the figure, it is divided into two runs 1 and 2. The labeling processor KLP writes a flag in the line buffer memory LBM until the last scan of the run reaches the last pixel, determines a temporary label, and connects all pixels of the run to all pixels of the run at the last pixel of the row. Is written in consideration of the temporary label.

As a result of scanning the run 1 shown in FIG. 9C and performing the above-described processing, the temporary labeling shown in FIG. The labeling of the temporary label “1” is performed on each pixel at the same time. This is because a labeling memory KLM, which will be described in detail later, is used as the memory. Subsequently, by scanning the run 2, the temporary labeling shown in FIG. Since Run 2 is connected to the temporary label “1” of Run 1, the temporary label “1” is simultaneously written to each pixel of Run 2 when the last pixel of Run 2 is scanned. By the labeling using the run as described above, the labeling to the step-like pixels shown in FIG. 9B can be performed by one kind of label “1”, and the number of temporary labels is reduced. In other words, the pixels are grouped into a single unit called a run, and processing is performed in units of a run, whereby the size of the labeling circuit can be reduced.

Next, details of the labeling process by the labeling processor KLP will be described. Labeling is performed by the window shown in FIG. 20 scanning each pixel along each line. By scanning this window along each run, a pixel of interest appears in the T (target) portion, a pixel located above the T portion appears in the a portion, and a pixel located on the right side of the T portion appears in the b portion. The next neighboring pixel appears. Hereafter, T, a,
b indicates the label value of the input image appearing in each part. When the window is scanning in the middle of the run, the value of the flag is output to the line buffer memory LBM as an output label, and when the window reaches the last pixel of the run, all the flagged pixels are output. The label is written.

The internal configuration of the KLP is shown in the block diagram of FIG. The KLP includes a selector 2101, a temporary label register Tml2102, a counter Cnt2103, a first comparison circuit 2104, and a second comparison circuit 2105. The first comparison circuit 2104 is provided with the label values T, a, and b, and performs a multi-value comparison of the input image. The comparison result is the select terminal sel1 of the selector 2101.
Given to. The value Mat (a) of the label matching memory LMM of the label value a, the value of the temporary label register Tml2102, and the value of the counter Cnt2103 are given to the second comparison circuit 2105, and at the same time, each value is the terminal A, B of the selector 2101. , C. The value of Tml2102 is determined by the output signal from selector 2101. The value of the flag FLAG stored in the line buffer memory LBM is given to the terminal D of the selector 2101.

The second comparison circuit 2105 compares these given values. The connection relationship between the labels is determined based on the comparison result, and the comparison result is output to the SELECT terminal sel0 of the selector 2101. The selector 2101 determines the matching address MAT ADDR and the matching data MAT based on the given values.
Outputs DATA to change the contents of the label matching memory LMM. At the same time, the selector 2101 outputs the value of the temporary label and the value of the output label (LABEL).

FIG. 22 to FIG. 27 show a flowchart of the window processing.

FIG. 22 is a flowchart showing a process of comparing and judging each value T, a, b in the first comparison circuit 2104. First,
It is determined whether the label value T of the pixel of interest is equal to 0 (step 2201). If T is 0, processing 1 described later is executed (step 2202). If T is not 0, the label value T is compared with the label value b (step 2203).
If the label value T is equal to the label value b, the label value T is compared with the label value a (step 2204). If the label value T is equal to the label value a, processing 2 is executed (step 2205). That is, the process 2 is a process executed when the values T, a, and b are equal. In this case, when each label value is expressed as ○, the window state is a state drawn adjacent to the illustrated processing box in step 2205.

If the label value T is not equal to the label value a, processing 3 is executed (step 2206). That is, the process 3 is a process executed when the label value T is equal to the label value b and the label value T is different from the label value a. in this case,
The label value T and the label value b are expressed as ○, and the label value a is expressed as ×
In this case, the window state is a state drawn adjacent to the illustrated processing box in step 2206. And
After the processing in step 2205 or 2206, a flag is set at the position of window T in line buffer memory LBM1603 (step 2207).

Further, even when the label value T is not equal to the label value b in step 2203, the label value T is compared with the label value a (step 2208). Label value T and label value a
Are equal to each other, processing 4 is executed (step 220).
9). That is, the process 4 is a process executed when the label T is equal to the label a and the label value T is different from the label value b. In this case, the label value T and the label value a
When the label value b is expressed as x, the window state is a state drawn adjacent to the illustrated processing box in step 2209.

If the label value T is not equal to the label value a, processing 5 is executed (step 2210). That is, the process 5 is a process executed when the label value T is different from the label value a and the label value T is different from the label value b. In this case, if the label value T is represented by ○ and the label value a and the label value b are represented by ×, the window state is set to step 2210.
Is drawn adjacent to the processing box shown in FIG.
Then, after the processing of step 2209 or step 2210, KL
Clear the temporary label register Tml in P (step 221)
1). This Tml is the T just before reaching the current window position.
The label for the pixel in the section is stored.

The flowcharts of FIGS. 23 to 27 described below show the flowcharts of the comparison determination processing from processing 1 to processing 5.

FIG. 23 shows a flowchart of the above-mentioned processing 1. Processing 1 ends without executing anything.

FIG. 24 shows a flowchart of the above-mentioned process 2. First, the label value of the previous pixel stored in the temporary label register Tm1 is compared with 0 (Step 2401). If the label value of Tml is equal to 0, the label matching memory (LMM) 1604 of the label value a is used. Is written to the temporary label register Tml2102 (step 2402). If the label value of Tml is not equal to 0, the label value of Tml and the counter C
The value is compared with the counter value of nt2103 (step 2403). The newest label value is stored in the counter 2103.

When the label value of the temporary label register Tml is equal to the counter value of the counter Cnt, the value Mat (a) of the label matching memory (LMM) of the label value a is written to Tml (step 2404). If the value of Tml is not equal to the value of the counter Cnt, the value of the LMM Mat (a) is compared with the value of Tml (step 2405). If the value of LMM Mat (a) is equal to the value of Tml, nothing is executed (step 240).
6). When the value of LMM Mat (a) and the value of Tml are not equal, the smaller value {Min (Tml, Mat (a))} of the value of Tml and the value of LMM Mat (a) is calculated. Write to Tml. Furthermore, the smaller value of the two, ｛Min (Tml, Mat
(A)) a label matching memory for a label value equal to the larger of the two (Step 2407).

FIG. 25 is a diagram showing a flowchart of the above-mentioned process 3. In the process 3, first, the value of the temporary label register Tml and 0
Are compared (step 2501). If the value of Tml is equal to 0, the counter value of the counter 2103 is written to Tml (step 2502). If the value of Tml is not equal to 0, nothing is executed (step 2503).

FIG. 26 is a diagram showing a flowchart of the above-mentioned process 4. First, the value of Tml is compared with 0 (step 260).
1). If the value of Tml is equal to 0, the LMM of the label value a
Value Mat (a) as the label value of the target area,
This Mat is used for all registers where the T section and flags are set.
Write (a) (step 2602). At this time, all the LBM flags are cleared. If the value of Tml is not equal to 0 in step 2601, the value of Tml is compared with the value of Cnt (step 2603). When the value of Tml is equal to the value of Cnt, the value of the LMM Mat (a) of the label value a is set as the label value of the target area, and the Mat (a) is set in all the registers where the flag is set in the T section. Write (Step 260
Four). At this time, all the LBM flags are cleared.

If the value of Tml is not equal to the value of Cnt, Tm
The value of l is compared with the value of Mat (a) (step 2605).
Then, when the value of Tml is equal to the value of Mat (a),
This Mat (a) is written to all the registers where the flag is set with the T part, using the LMM value Mat (a) of the label value a as the label value of the target area (step 2606). At this time, all the LBM flags are cleared. Also, the value of Tml
If the value of Mat (a) is not equal, the smaller value of the value of Tml and the value of LMM Mat (a) of the label value a ｛Min (Tm
l, Mat (a))} as the label value of the target area, and writes it to all the registers in which the T section and the flag are set. At this time, all the LBM flags are cleared. Furthermore, the smaller value {Min (Tml, Mat (a))} of the value of Tml and the value of LMM Mat (a) is matched with the label value of the label value equal to the larger value of both. memory (Step 2607).

FIG. 27 is a diagram showing a flowchart of the aforementioned process 5. First, the value of the temporary label register Tml is compared with 0 (step 2701). Cut if the value of Tml is equal to 0
Is used as the label value of the target area, and the value of Cnt is written to the T section and all the registers in which the flags are set. At this time, all the LBM flags are cleared. In addition, the value of Cnt is replaced by the label value Mat (Cnt) equal to the value of Cnt.
And increment the value of Cnt by one (step 2702).

If the value of Tml is not equal to 0, the value of Tml is written to all the registers in which the T section and the flag are set, using the value of Tml as the label value of the target area (step 2703). At this time, all the flags of the LBM are cleared. Next, the value of Tml is compared with the value of Cnt (step 27).
04). If the value of Tml is equal to the value of Cnt, the value of Cnt is written to Mat (Cnt) of the label value equal to the value of Cnt. Further, the value of Cnt is incremented by one (step 2705). If the value of Tml is not equal to the value of Cnt, nothing is executed (step 2706).

Next, the labeling memory KLM used for the line buffer memory LBM and the label matching memory LMM will be described. Until now, data could only be written to one internal register by one address specification.
However, by using the KLM described below, data can be written into a plurality of internal registers at once. For this reason, this labeling memory KLM is used for a line register (LBM) for run processing, a label matching memory (LMM) that does not require label integration, a label image memory (LABELM) for one scan, and a label matching memory that can perform label matching. You can do it.

LKM is composed of multiple registers.
The figure shows a block configuration of one of these registers. This block is one unit of the KLM configuration. Each register 2801 is connected to a comparator 2802 as a pair. This comparator 2802 includes:
Output data DATA from the register 2801 and information COM to be compared with the output data are provided. Comparator 2802 compares the applied data and outputs the comparison result to AND circuit 2803. The output of the address decoder circuit 2804 is also supplied to the AND circuit 2803. The AND circuit 2803 includes a comparator 2802 or a decoder circuit 2804.
If one of the two outputs a signal, the OR circuit 2805
Output the signal.

The OR circuit 2805 is supplied with a write signal WR from the CPU.
01 is supplied with an enable signal. That is, when the address is selected by the address decoder circuit 2804 or the comparison result in the comparator circuit 2802 matches, the write signal
Data is written to the register 2801 in synchronization with WR. The addressing to each decoder circuit 2804 and the comparison judgment in each comparator circuit 2802 are all executed simultaneously. Therefore, the contents of a plurality of registers constituting the KLM can be rewritten simultaneously by one addressing or one data comparison judgment.

The temporary label obtained by the labeling process using the run described above becomes a discontinuous value when the label integration is performed.
The contents of the label matching memory LMM at this time are shown in FIG. Labels 1, 3, and 6 are stored as data corresponding to the pixels at addresses 1 to 10, respectively. Since this label value is a discontinuous value, it is converted into a label having a continuous value shown in FIG. 3C by the feature extraction processor KLC shown in FIG. That is, the label value stored in the label matching memory LMM constituted by the KLM is a continuous value of 1, 2, and 3.

More specifically, the labeling processor KLC generates an address of the LMM and takes in the data indicated by the address.
Subsequently, the fetched data is compared with the address, and if the values are the same, new data is output to the LMM to rewrite the label value. If the values are different, the next address is output to the LMM, and the next address is compared with the data. Thereafter, by repeatedly executing this process,
The discontinuous label values shown in FIG. 29 (a) are shown in FIG. 29 (c).
Is converted to the continuous label value shown in.

Specifically, the data (1) at address 1 in FIG.
Is the same as the address, so KLC outputs 1 as new data, and the same data (1) as the data at address 1
Is rewritten with new data 1 at addresses 2, 4, 5, and 7 having. In the case shown in the figure, the old data and the new data happen to be the same 1, so that there is no change in the data of the corresponding address in FIGS. Next, the data (1) at address 2 is compared with the address. Since the address and the data are different, the next address 3
Occurs. Then, data (3) at address 3 is compared with the address. Since the address and the data are the same, 2 is output as new data. Since the LMM is constituted by the KLM, the data at addresses 9 and 10 having the same data (3) as the data at address 3 are all rewritten to 2 at the same time as shown in FIG. Next, KLC
Generates a new address 4. Since the address and the data are different, the address 5 is generated next, and thereafter, the same processing as described above is repeated. As a result, discontinuous values are converted to continuous values.

FIG. 30 is a block diagram showing the internal configuration of this feature extraction processor KLC. By using this KLC,
Pre-processing of temporary labels generated by primary labeling is performed. At the time of the secondary labeling, at the same time as the labeling, the calculation of the area of the same label area, the X of the same label area are performed.
The KLC executes the calculation of the sum of the direction addresses and the calculation of the sum of the Y-direction addresses of the same label area.

KLC consists of a +1 adder 3001 and two adders 3002, 3003
And a comparator 3004 and two counters 3005 and 3006. When there is a pixel input having the same label value, the +1 adder 3001 counts up the count number one by one, calculates the area of the same label area, and outputs the result as Size. The X direction address value and the Y direction address value are input to the adders 3002 and 3003. And
When there is a pixel input of the same label value, the address value is added for each direction, and the sum of the address values of the same label area is calculated for each address direction. Each sum is
Output as X ADDR and Y ADDR. By dividing the sum of the addresses in each direction by the area of the same label area,
The center of gravity in each direction can be obtained. Then, the center of gravity of each identical label region is obtained, and the region having the center of gravity above the horizon is removed as not being an effective region for extracting a road candidate region.

The comparator 3004 and the counters 3005 and 3006 are used for preprocessing of temporary labels generated at the time of primary labeling, that is, processing for converting discontinuous label values to continuous label values. The clock signal CLK is input to the counters 3005 and 3006, and the output of the counter 3005 is a label matching memory LMM.
Becomes the matching address MAT ADDR output to This address is also given to the comparator 3004 at the same time. The output of the counter 3006 is data MAT DATA output to the label matching memory LMM. The comparator 3004 compares the given address and data as described above, and outputs a signal to the counter 3006 when the respective values of the address and data match. When this signal is input, the counter 3006 outputs the current counter value to Mat DATA and sets the value to 1
Count up.

The processing time of the labeling process described above requires the sum of the raster scanning time of two scans performed along each run and the label integration time. But the 16th
By configuring the label memory LABELM1606 shown in the figure using the above-described labeling memory KLM, the labeling process can be executed in one scan. That is, as shown in FIG. 31, when a multi-valued input image 3101 is captured, the labeling processor KLP3102 executes a labeling process using the line register 3103 and the label matching memory 3104 in the same manner as described above. The label value of each pixel obtained by this labeling is stored in the label image memory at the time of scanning the last pixel of each run.
It is written to 3105 as it is. Since label integration is not performed, the value of each label is stored as a discontinuous value.

As described above, the labeling memory KLM has a pair of comparators connected to each internal register, and new data is written at a time to the register whose comparison result matches and the register whose address is selected. It is. This KLM feature allows one-scan labeling. The labeling processor KLP is a hardware version of the algorithm of the present system, and enables high-speed labeling.

With this one-scan labeling method, it is possible to reduce the labeling processing time to half or less than before. For example, when the device is operating with a clock signal of 8 MHz, the labeling processing time up to now required 66 msec for scanning two windows and the time required for label integration. However, according to the one-scan method, the labeling processing can be performed in 33 msec which is 1/2 or less.

In addition, normal pixel scanning is performed from the upper left to the lower right of the screen of the input image, but in image processing for extracting a road region,
The target image is located below the image. Therefore, there is a possibility that the memory for storing the temporary label overflows when the target image is scanned. For this reason, scanning is performed from the lower right to the upper left of the screen, and the target image is scanned first so that the target image can always be obtained. That is, if the target image is acquired first, even if a memory overflow occurs, the image scanning at the time of the overflow becomes a scan for an unnecessary image portion, and the target image can always be acquired.

Next, the “method of merging a plurality of regions” obtained by dividing an image will be described.

A road candidate area is obtained by "a method of extracting a traveling course by repeated threshold processing using a template image", and a low-luminance area is extracted by "an extraction means of a shadow or a high-luminance portion on the traveling course focusing on a difference in brightness". Shadows and high-luminance areas. Based on this road candidate area, a low-luminance area and a high-luminance area strongly connected to the road candidate area are merged into a road candidate area, treated as one area, and regarded as a traveling course. For this purpose, the common boundary length adjacent to each area and the perimeter of each area are determined, and the relationship between the connected areas is described. The description of the relationship between the regions indicates the merging relationship, and the same result as that of the merging process of the regions can be obtained without performing the label change operation as in the related art. Note that the common boundary length is determined based on the length of the side of the pixel adjacent to each area, and the perimeter is determined based on the length of the side of the outermost pixel of each area.

The algorithm of this method is shown below. This method is of two types. First, there is an inverted L-shaped mask scanning method for scanning an inverted L-shaped mask. This method is a simple algorithm suitable for hardware implementation. Secondly, there is an area boundary search method for locally searching the boundary of an area. This method searches only the boundary of a necessary area, so that it can be processed with a small memory and is effective.

The first method, the inverted L-shaped mask scanning method, is executed by scanning the inverted L-shaped mask shown in FIG. 32 from left to right and from top to bottom of the label image. The x pixel appearing in the illustrated mask is a pixel of interest, the a pixel is a pixel located above the pixel of interest x, and the b pixel is a pixel located to the left of the pixel of interest x. For example, assume the pixel area shown in FIG. The □ shown in the figure represents one pixel, and the numerical value in the □ represents the label value of the pixel. In the case of this example, the area of label value 1 and the label value 4
Are adjacent to each other. Although not shown in the figure, the label value of the background area is 0.

The perimeter length ob of each area is obtained by scanning the inverted L-shaped mask along the row of each area, that is, along the run. Specifically, the perimeter ob1 of the area of the label value 1 is obtained as follows. First, the x pixel of the inverted L-shaped mask is matched with the pixel located at the left end of the uppermost run. In this case, since the pixel a and the pixel b are in the background area and the label value is 0, it can be understood that two sides of this pixel are located around the area of the label value 1. Therefore, 2 is added to the counter for counting the perimeter ob1. Next, the inverted L-shaped mask is scanned to the right, and the x pixel is adjusted to the pixel on the right. In this case, the pixel a is located in the background area, and the pixel b is located in a pixel in the main area having a label value of 1 with the previous mask. Therefore, it is found that one side of this pixel is located around the main area of the label value 1, and 1 is added to the counter. By scanning the area of the label value 1 along the run with the inverted L-shaped mask in this manner, the perimeter ob1 becomes 18. The unit of the perimeter ob is the number of sides of the pixel.

The perimeter ob4 of the area of the label value 4 is also obtained in the same manner as the above perimeter ob1, and its value is 20.

Further, the common boundary length cb between the area of the label value 1 and the area of the label value 4 can be obtained as follows. The common boundary length cb is different between a common boundary length cb14 with the label value 4 area viewed from the label value 1 area and a common boundary length cb41 with the label value 1 area viewed from the label value 4 area. There are cases. In other words, the mask has an inverted L-shape, and only two neighboring pixels of each pixel are taken into account, causing a difference in the common boundary length. However, there is no problem in the merging processing of a plurality of regions. It should be noted that if the mask to be scanned has a cross shape that looks at the vicinity of 4 of the pixel of interest, there is no difference in the common boundary length.

Common boundary length with label value 4 as seen from the area of label value 1
cb14 is obtained by mask scanning when obtaining the perimeter ob1. This common boundary length cb14 becomes zero. That is,
This is because even if the inverted L-shaped mask is scanned for each run in the area with the label value 1, the pixel with the label value 4 does not appear in the a-pixel and the b-pixel.

The common boundary length cb41 with the label value 1 area viewed from the label value 4 area can be obtained by mask scanning when obtaining the peripheral length ob4. This common boundary length cb41 is 3. That is, when the mask is located at the first pixel of the second and third runs in the area with the label value 4, the pixel with the label value 1 appears at the b pixel. Therefore, the value of the counter that counts the common boundary length cb41 becomes 2 by the mask scanning for the runs up to the third stage. Further, when the mask is located at the head position of the fourth run, a pixel having a label value of 1 appears at the a pixel of the mask. Therefore, 1 is added to the counter, the integrated value of the counter becomes 3, and the common boundary length cb41 becomes 3. The unit of the common boundary length cb is the number of sides of the pixel.

Next, the values of the common boundary length between the regions and the perimeter of each region thus determined are stored in the boundary length matrix between labels shown in FIG. The numbers 0, 1, 2,..., J,... K assigned to each column of the matrix and the numbers 0, 1, 2,. The numeric value mij stored in the location specified by the numeric value in each column and each row
Is a common boundary length with the area of the label value j as viewed from the area of the label value i. The numerical value ti stored in the (k + 1) th column is the perimeter of the area of the label value i.

Taking the pixel area shown in FIG. 33 as an example, label value 1
Since the value of the common boundary length cb14 with respect to the area of the label value 4 as viewed from the area of 0 is 0, 0 is stored in the memory located in the first row and the fourth column of the same matrix. Further, since the value of the common boundary length cb41 with the area of the label value 1 as viewed from the area of the label value 4 is 3, 3 is stored in the memory located at the 4th row and the 1st column of the same matrix.

After a matrix is created by calculating the perimeter of each region and the common boundary length between the regions, the connectivity between the regions is calculated. This connection degree is obtained by the following equation based on the common boundary length and the perimeter.

Degree of connection = X / min (A, B) where A is the perimeter of the area of the label value i, B is the perimeter of the area of the label value j, and min (A, B) is each of A and B Represents the smaller of the numbers. X is the average value of the common boundary length between the respective areas represented by the following equation.

X = (X1 + X2) / 2 X1 is a common boundary length with the area of the label value j viewed from the area of the label value i, and X2 is the label value i viewed from the area of the label value j.
Is the common boundary length with the region of. The above expression for X is X1
This is effective when both X2 and X2 are not 0 (X1 ≠ 0, X2 ≠ 0). When X2 is 0 (X2 = 0), X is represented by the following equation.

 X = X1 When X1 is 0 (X1 = 0), X is represented by the following equation.

X = X2 If the calculated connectivity is equal to or greater than a predetermined threshold, the connection between the area of the label value i and the area of the label value j is strong, and the respective areas are in a relationship to be merged. If the degree of connection is equal to or less than the predetermined threshold value, the connection between the area of the label value i and the area of the label value j is weak, and the areas do not have to be merged. This merging relationship is described in the connection label column of the feature amount list shown in FIG.

The label No. of the list is a label value, and the area and the center of gravity are the area and the center of gravity of the area of the label value. The area and the center of gravity are obtained at the time of the labeling process described above.
The circumscribed rectangle is a rectangle that surrounds all the pixels in the area of the label value. The circumscribed rectangle is specified by the coordinates tL (x, y) of the upper left vertex and the coordinates bR (x, y) of the lower right vertex among the four vertices that are the intersections of the sides. Each coordinate is obtained in the above-described boundary search process. Further, the label No. described in the connection label column indicates the label value of an area which is to be merged with the area of the label value. When there is an area to be further merged with the area described in this column, as shown in the figure, the connection labels are further connected by a pointer.

Each feature described in the list is for each area shown in FIG. Label values l1 to l4 are assigned to the respective regions, and G1 to G4 indicate the positions of the centers of gravity of the respective regions. A circumscribed rectangle is drawn in each area. A contact point between one side on the rectangle and the region is represented as coordinates st (x, y), and is obtained by performing a horizontal operation from the region start point tL (x, y). The coordinates st are used in a method described later.

Since the area of the label value l1 and the area of the label value l2 are to be merged, the label value l2 is described in the connection label column of the label No. l1 line of the feature amount list. Furthermore, since the area of the label value l1 has a relationship to be merged with the area of the label value l3 and the area of the label value l4,
The connection label of l4 is connected by the pointer. Further, the connection relation between the respective areas is described in the connection label column of each row of the label Nos. L2 to l4 in the same way.

 Next, an area boundary search method will be described.

First, a rectangle circumscribing the target area is drawn, and a search start pixel located at coordinates st (x, y) where the circumscribed rectangle and the area are tangent is determined. And 8 of this search start pixel
Investigate the pixels located in the vicinity. Each reference pixel located in the vicinity of No. 8 is assigned a position No. as shown in FIG. That is, the No. located above the pixel of interest is set to 0,
Number sequentially in clockwise order. The coordinates of the pixel to be searched next to the search start pixel are determined using the pixel reference tables shown in FIGS. 38 (a) and (b). The table shown in FIG. 6A is a table referred to when a pixel search is performed clockwise (clockwise), and the table shown in FIG. This is a table that is referred to when executing clockwise. Numerals 0 to 7 between parentheses {} in the table correspond to the position numbers of the reference pixels in FIG.

The pixel to be searched next is determined by obtaining the reference pixel position number corresponding to the current pixel position with respect to the previous pixel position, and referring to the column number of the row number of the table corresponding to this position number. Is done. A method for determining a search pixel using the pixel reference table will be specifically described below. Assume that the boundary of the pixel area is formed as shown in FIG. In the figure, the hatched squares indicate pixels whose label value is not 0. The previously searched pixel is indicated by the symbol △,
The currently searched pixel is indicated by a symbol 、, and the next pixel to be searched is indicated by a symbol ◎.

The current pixel position △ with respect to the previous pixel position 、 is the 37th
The reference pixel position No. in the figure corresponds to 1. If the pixel near the boundary of the area is searched clockwise, the pixel reference table refers to the table in FIG. 38 (a). Since the reference pixel position No. is 1, the row No. is 1 and the column N
o. Refer to {7,0,1,2,3,4,5,6}. The row No. is 0 in the top row, and is 1 to 7 in order. Also columns
The number of the pixel to be referred is described in the No., and the modulo calculation in the normal address calculation can be avoided by referring to the eight neighborhoods of the pixel in this order.

That is, since the column No. starts from 7, the next pixel to be searched is a pixel ◎ which is located diagonally upper left of the current position ◎.
become. Pixel ◎ is a background area, and the label value is 0, so the next pixel is searched according to the column No. That is, since the next reference pixel position No. is 0, a pixel located above the current position ○ is searched. Since this pixel is also in the background area, a search is made for a pixel whose reference pixel position No. is 1 according to the column No. Since this pixel has a label value and is not 0, the reference of the next pixel to be searched is set to the pixel at this position No. 1, and the search for the eight neighboring pixels is performed in the same manner as described above.
Then, the above processing is performed for each pixel along the boundary of the area, and the same processing is repeated for each pixel until the pixel returns to the search start pixel.

In addition, in this search processing, by performing processing according to the following rules, the perimeter of the own area of the label value i is obtained.
common boundary length cbi between obi and the adjacent region of label value j
j can be obtained. That is, if the label value j exists in the area adjacent to the four neighborhoods of the current position, up, down, left, and right, 1 is added to the counter that counts the common boundary length cbij.
At the same time, the counter that counts the obi
Is added. If there is no pixel having the same label value as its own label value i in the vicinity of 4 at the current position, 1 is added to the count of its own perimeter.

Next, based on the determined perimeter and common boundary length, the inverse L
The connectivity between the respective regions is obtained in the same manner as the calculation in the character mask operation formula method. At the same time, a feature amount list is created in the same manner as described above. The connection label between the regions is described in the connection label of the feature amount list, and the merging relationship of a plurality of regions is determined. That is, even with this method, the merging relationship of each area can be determined without performing a label change operation of each area. The above boundary search was performed clockwise. However, when the boundary search is performed counterclockwise, the clockwise and counterclockwise search is performed by using the counterclockwise pixel reference table shown in FIG. 38 (b). Processing can be performed in a similar manner.

By using the pixel reference table in this manner, a fast boundary search can be performed. In other words, it is not necessary to check all the pixels in the vicinity of the pixel 8 serving as the reference for the search.
Also, only the pixels near the boundary of the region need to be searched,
There is no need to search for all the pixels in the area.
Therefore, processing time is reduced.

According to each of the methods described above, the merging relationship between the divided areas can be determined without performing the label replacement operation as in the related art. In other words, the road candidate area obtained by the “extraction method of the traveling course by the repetitive threshold value processing using the template image” and the “extraction means of the shadow and the high luminance portion on the traveling course focusing on the difference in brightness” are used. The obtained low-brightness and high-brightness areas are merged, and a real-world running course area can be obtained.

Next, the "means for determining the travelable range" will be described. This means is based on “a method of extracting a running course by repeated threshold processing using template images”, “threshold setting means based on the shape of the feature histogram in repeated threshold processing”, and “focusing on differences in brightness. The final travelable range, that is, road end coordinates, is obtained from the traveling course image obtained by the “extraction means for shadows and high-luminance portions on the traveling course”.

FIG. 40 is a flowchart showing a flow of the entire processing of the traveling course recognition system according to the present technique. First, the image information of the traveling course is captured by the color camera input device, and the captured R, G, B images are converted into the brightness I,
The image is converted into each image of the saturation S. Based on each of the converted images, a histogram using the brightness I as a feature amount and a histogram using the saturation S as a feature amount are created in the color processing apparatus as described above (step 4001). Next, based on each of the created histograms, a road candidate area is extracted by a “running course extraction method using repeated threshold processing using a template image”, and small areas are removed (step 40).
02).

Next, a point sequence at the road end of the extracted road candidate area is evaluated by this method (step 4003). Then, based on the evaluation result of the point sequence at the road end, it is determined whether or not the extracted road candidate area is a monotone road (step 4004). If the road is a road-like monotone road, the relation between the plurality of obtained point sequences is further compared and evaluated (step 4005). Then, the evaluation result is finally output (step 400).
6). Thereafter, it is assumed that a dot sequence for the image input this time has been obtained, and the process for the next image is executed.

If the result of the determination in step 4004 is that the road-likeness is low and the road is not a monotonous road, a low-luminance area is present due to “extraction means for shadows and high-luminance parts on the traveling course focusing on the difference in brightness”. If this is the case, it is extracted (step 4007). Then, the obtained low-luminance area and the road candidate area are merged, and the point sequence at the road end is evaluated by this method (step 4008). Based on the evaluation result of the point sequence at the road end, it is determined whether or not the extracted road candidate area is a monotone road (step 4009). If the road is a monotone road, execute the processing after step 4005,
The processing shifts to processing for the next image.

If the road is not a monotonous road, then, “extraction means for shadows and high-luminance parts on the traveling course focusing on the difference in brightness”
, If there is a high-luminance area, extract it (step
4010). Then, the obtained high-luminance area and the road candidate area are merged, and the point sequence at the road end is evaluated by this method (step 4011). Based on the evaluation result of the point sequence at the road end, it is determined whether or not the extracted road candidate area is a monotone road (step
4012). If the road is a monotone road, the processing after step 4005 is executed, and the processing shifts to the processing for the next image. If the road is not a monotone road, the low-luminance area and the high-luminance area are merged with the road candidate area, and the point sequence at the road end obtained from the merged area of the three areas is evaluated (step 4013). Thereafter, the process of step 4005 is executed, and the process proceeds to the next image.

Next, a method for obtaining the point sequence coordinates of the road end of the road candidate area will be described below.

For example, assume that the image area shown in FIG. 41 is obtained. The origin is at the upper left of the figure, the x coordinate is positive in the horizontal direction to the right, and the y coordinate is positive in the vertical direction downward. There are cuts in the road area shown in this image, and places where image information could not be obtained due to noise or the like are scattered. It is assumed that a labeling process is performed on each pixel of the image shown in FIG. 42, and a pixel region obtained as a result of this process is as shown in FIG. The hatched portion in the figure is a pixel having a certain label value instead of a label value of zero. Note that pixels without oblique lines are pixels located in the background area where the label value is 0. A frame 42 surrounding 3 × 3 pixels
01 is a window W, and the number of hatched pixels (pixels having pixel values other than 0) existing in the window W is a histogram value. Window W when positioned as shown
The value of the histogram according to is 5. Although the illustrated window W is a 3 × 3 window, it may be a window of another size such as 5 × 5.

First, the window W is horizontally scanned from the center of the image to the left. The window W is scanned from the center of the image only at the start of the processing. The value of the histogram of the window W at each position is determined while moving. When the window W is moved while monitoring the histogram value, and a histogram value equal to or less than a predetermined threshold value is continuously obtained, that is, when the density of hatched pixels in the window W decreases, the window W It is determined that the vehicle is out of the road area. Then, the position of the first window W where the histogram value becomes equal to or less than the predetermined threshold value is set as the point sequence coordinates XL of the left road end. Next, the window W is horizontally scanned rightward, and when the histogram value in the window W continuously falls below a predetermined threshold value, the position of the first window W where the histogram value changes is set to the right side. Point sequence coordinates XR at the road edge.

Next, the Y coordinate at which the window W is located is reduced by one,
The window W is moved upward of the image to shift four horizontal scanning positions. Then, the center position of the road area is calculated from the road end coordinates XL and XR obtained by the first horizontal scanning by the equation (XR−X
L) / 2 is calculated. The center position is set as the scanning start position of the window W, and the window W is scanned left and right. Also in this scanning, the histogram value in the window W is monitored in the same manner as described above, and the position where the histogram value changes is set as the coordinates of the road edge. Hereinafter, by repeating the same processing until the Y coordinate of the horizontal scanning position becomes the horizon position, a series of point sequence coordinates of the left and right road edges can be obtained.

In the scanning of the window W, the histogram value obtained from the window W at the scanning start position is equal to or less than the predetermined threshold value from the beginning, and the histogram value obtained by scanning the window W to the left side is continuously. If the distance is equal to or smaller than the predetermined threshold value, the center position becomes the road end coordinates. However, in this case, the scanning direction of the window W is changed to the opposite right side. When the histogram values equal to or more than the predetermined threshold value are continuously obtained, the position of the window W where the histogram value equal to or more than the predetermined threshold value is obtained first is set as the coordinates of the left road edge. . The coordinates of the right side road edge can be found by scanning the window W further rightward and finding the position where the histogram value changes.

Next, since the point sequence coordinates of the road end obtained in this way exist near the boundary of the area, the boundary line obtained by connecting the points is not uniformly smooth. For this reason, the angles formed by each point located before and after a certain point and this one point are calculated for all the points in the point sequence, and this variance value is used as a reference for smoothness. When the obtained angle is an acute angle, the point is removed for calculation. After smoothing the point sequence coordinates of the left and right road edges in this way, these point sequence coordinates are projectively transformed into the real space. This projective transformation is performed based on the depression angle, height, and focal length of the color camera attached to the autonomous vehicle.

Further, in the point sequence after the projective transformation, the distance between the left and right point sequences, that is, the road width, is obtained by setting the left and right point sequences as a set. Then, the number of points where the road width becomes narrower than the vehicle body width of the traveling vehicle is counted, and the points of this narrow width with respect to all points in the point sequence are calculated. When the ratio of the points of the narrow width to the total number of points is small, the points of the narrow width are not suitable as the road end points, and thus these points are removed.

In evaluating the point sequence at the road edge, the left and right pairs that have a small difference from the road edge determined by the first set of left and right edges are counted. When the ratio of the left and right pairs with a small difference to the total pair is large, that is, when the left and right road end rows are parallel to some extent, the evaluation as road-likeness is the highest, and it is judged as a monotone road. Is done. When it is determined that the road is a monotonous road, the road edge can be obtained without performing the processing of steps 4003, 4008, and 4011 in the flowchart shown in FIG.

Next, a description will be given of a means for correcting a point sequence at the left and right ends of the road when the self-contained traveling vehicle is extremely close to the road edge. FIGS. 43 (a) to (d) show a process in which an autonomous vehicle approaches a road edge. FIG. 3A shows a case where the autonomous traveling vehicle is located just before a sharp curve. In this case, the left road end L and the right road end R are captured in the field of view by the color camera attached to the autonomous vehicle. When the curvature of the road curve is tight, the autonomous vehicle located in FIG. 11A moves to the position shown in FIG. In this case, the right road edge R disappears from the field of view of the camera.

Further, the autonomous vehicle moves to the position shown in FIG. 9C, and the ratio of the area outside the road to the template gradually increases. The shaded area in the figure indicates the area outside the road. Further, the autonomous vehicle moves to the position shown in FIG. In this case, the left side of the road edge on the left side and the left side boundary of the area outside the road are captured in the field of view of the camera. In addition, the ratio of the area outside the road to the template image increases, and the area outside the road is mistaken for the traveling course. As a result, the left road edge is erroneously recognized as the right road edge R, and the left boundary of the area outside the road is erroneously recognized as the left road edge L.

However, the point sequence at the road end obtained from the sequentially captured image information does not become the point sequence on the right side in the previous image that is extremely left on the current image. Conversely, the point sequence on the left side in the previous image does not become a point sequence extremely located on the right side in the current image. Therefore, the point sequence information obtained from the image information one processing cycle ago is stored and held, and the distance between the point sequence obtained from the current image information and the point sequence obtained from the previous image information will be described later. calculate. Then, for example, if the distance between the right-hand point sequence obtained at the last time and the right-hand point sequence obtained this time is far away, and the last right-point sequence is close to the current left-point sequence,
It is determined that the positions of the left and right road edges have been reversed. Then, the right side and the left side of the road edge obtained this time are switched.

This will be described with reference to the image shown in FIG. The image in FIG. 3C is the image obtained last time, and the point sequence at the left side of the road is expressed as Li-1. The image shown in FIG. 3D is the image obtained this time, and the point sequence at the left road edge that is erroneously recognized is expressed as Li, and the point sequence at the right road edge that is erroneously recognized is expressed as Ri. The determination as to whether the point sequence at the left and right road ends is reversed is made by determining which of the current point sequence Li and Ri is closer to the previous point sequence Li-1. The previous point sequence Li-1 is the current point sequence
If it is close to Ri, it is determined that the point sequence at the left and right road edges has been reversed.

Generally, the closeness of the point sequence is determined as follows.
That is, the left point sequence obtained from this image is one group,
The point sequence on the right side is regarded as another group, and it is determined by judging which of the groups the previous point sequence on the left or right side is closer to. The determination as to which of these groups is closer is determined by Mahalanobis' generalized distance, which will be described in detail below. FIG. 44 is a diagram for explaining the Mahalanobis' generalized distance.

The point of the left point sequence Li-1 (or the right point sequence Ri-1) based on the previous image information is (x, y), the point of the left point sequence Li based on the current image information is (x1j, y1j), and the current image The point of the right point sequence Ri based on the information is (x2j, y2j). The point in the previous point sequence (x,
y) is closer to the current point sequence Li or Ri
The Mahalanobis' generalized distance is calculated using the point sequence Li-1 (or the point sequence Ri-
The calculation is performed as follows for all the points in 1), and it is assumed that points having short distances are closer to the point sequence to which many points belong. When the point sequence obtained from the current image is a straight line, the distance to this straight line is calculated.

First, for each point on the point sequence Li, the average value μ11 of the x-direction component, the average value μ12 of the y-direction component, and the variance σ11, y of the x-direction component
A variance value σ12 of the directional component, a covariance value σ112 of each of the x and y directions and a correlation coefficient ρ1 are obtained. Here, ni is the number of points located on the point sequence Li. In addition, each point on the point sequence Ri can also be obtained by an equation similar to the following equation.

μ11 = Σx1j / n1, μ12 = Σy1j / n1 σ11 ² = Σ (x1j-μ11) ² / (n1-1) σ12 ² = Σ (y1j-μ12) ² / (n1-1) σ112 = Σ (x1j-μ11 ) (Y1j−μ12) / (n1 / 1) ρ1 = σ112 / σ11 × σ12 Based on these numerical values, the point (x, y) of the previous point sequence and the point (x1j, y1j) of the current point sequence Li The Mahalanobis' general distance D1 between and can be obtained by the following equation.

D1 ² = {[(x−μ11) / σ11] ² + [(y−μ12) / σ12] ² −2ρ1 [(x−μ11) / σ11] × [(y−μ12) / σ11]} / (1 −ρ1) ² Similarly, the point (x, y) of the previous point sequence and the current point sequence Ri
The point (x2j, y2j) the Mahalanobis distance D ² between the can be calculated by the following equation.

D2 ² = {[(x−μ21) / σ21] ² + [(y−μ22) / σ22] ² −2ρ2 [(x−μ21) / σ21] × [(y−μ22) / σ22]} / (1 −ρ2) ² As a result of calculating each general distance based on the above equations, D1 ²
> If the D2 ² is satisfied, the last point (x, y) will be close to the point sequence Ri of this right roadside. Also, D1 ² > D2 ²
Does not hold, on the contrary, the previous point (x, y) is close to the current point sequence Li on the left side of the road.

As described above, even if the road candidate area obtained as a result of the image division processing becomes an area with an unclear contour due to noise or the like, the end point of the road is determined based on the change in the histogram value of the window according to the present method. Can be obtained reliably. This is because the present method searches for the end point based on the region, and according to the present method, the left and right ends of the region can be clearly distinguished.

In addition, the low-luminance area obtained by “extraction means for shadows and high-luminance areas on the traveling course focusing on the difference in brightness” may be located outside the road when the traveling course is entirely dark. is there. In such a case, the low luminance area outside the road and the road area may be merged, and the shape of the contour of the road candidate area becomes complicated. However, by smoothing the obtained point sequence data at the road edge according to the present method, the area of a complicated contour is globally evaluated, the low luminance area as described above is ignored, and the Thus, information on the road edge without any trouble can be obtained.

Also, according to the “running course extraction method using repeated threshold processing using a template image”, when an autonomous vehicle is extremely close to the left or right of a road with a sharp curvature,
In some cases, the template goes out of the road and regards it as a road. In this case, the recognition of the left and right of the road edge is erroneous. However, according to the above-described method, even if the data at the right and left ends of the road is erroneously recognized, this recognition is immediately corrected, and highly reliable road edge information can always be obtained.

Next, a description will be given of a feature of the present invention, that is, a “method of internally expressing traveling courses of various shapes”, that is, a method of structuring and expressing various obtained point sequences of road edges.

This method is based on the method of extracting the running course image obtained by the "running course extraction means by repeated threshold processing using template images" and "extracting shadows and high-luminance parts from the running course focusing on the difference in brightness." Based on the image obtained by the “means” and the feature list obtained by the “means of merging multiple areas”,
By creating a structure of the boundary point of the area while detecting the label displacement part of the image, attaching a link in the front and back direction and adding the attribute of the boundary, the travel course is expressed inside the computer, and the effective This is for obtaining a point sequence group. Using the feature list showing the description of the areas created by the "means for merging a plurality of areas", the end points of the area can be obtained at high speed, and the road end, the roadside end, the branch road, and the junction flow path are obtained. Is represented by a structured list inside the computer.

FIG. 1 is a flowchart showing the outline of the road structuring process.

First, a list in the y direction of the area boundary point structure is created at the time of initial setting of each hardware such as a color camera input device (step 4501). Next, a change in the x direction of the label value attached to each area is detected to create a list in the x direction, and a link is made in the x direction at each boundary point (step 4502). Then, the attribute is given to the left end and the right end of the area in consideration of the connection label of the feature amount list of each area, and the left and right ends of each area are classified (step 4503). Next, while adding attributes such as holes to be described later to each boundary point,
The respective boundary points in the front-rear direction (y direction) of the screen are associated with each other and linked in the y direction of each boundary point (step 45).
04). In addition, there is a case where the road area branches or merges. In each of these cases, the continuity of the boundary of each area is determined, and the attribute assigned to each boundary point is changed (step 4505). Thereafter, the start position of the point sequence at the boundary point of each area is selected, and an effective point sequence is detected (step 4506).

 The following processes will be described in detail below.

The processing in step 4501 will be described with reference to FIG.

FIG. 1A is a diagram showing the principle of projective transformation of a road surface (assumed as a horizontal plane) in real space onto an image. The road surface 4601 is sectioned at equal intervals (l), and the road area and each dividing line on the road are projected on the image 4602 toward a distant point, and a viewport of a color camera is obtained.
FIG. 2B is a diagram showing details of the image obtained in this manner. On the image, the boundary line 4603 and the division line 4604 of the road area are projectively transformed. Although the dividing lines in the real space have an equal interval of l, the intervals on the image become narrower as the distance from the road area increases. The y-coordinate at which each section line 4604 is located is determined, and a region boundary point structure 4605 in the y-direction shown in FIG.

Each frame of the area boundary point structure 4605 corresponds to each boundary point located at the intersection of the leftmost boundary line of the road area boundary line 4603 and the division line 4604. In each frame, a mutual connection relationship is represented by a pointer indicated by an illustrated arrow. In each of these frames, the features of each boundary point are described as a list 4606 shown in FIG.
First, each value of the obtained y coordinate is described in a list 4606, and y
A list of directions is created. In this list 4606, in addition to the above-mentioned y-coordinate, the following characteristics of each boundary point are described.

That is, the x coordinate, the attribute representing the point sequence starting point iD, the attribute 2 representing the distance between adjacent points in the real space, the pointer to the right indicating the connection relationship between the boundary point of interest and the boundary point on the right, the left Pointer to the left indicating the connection with the next boundary point, pointer to the front indicating the connection with the boundary point in front of the screen in the y direction, and Next indicating the connection with the next boundary point behind the screen Is described.

Next, the process of step 4502 will be described with reference to FIG.

An image 4701 shown in FIG. 11A is an image obtained by performing a labeling process on a captured image. By this labeling process, a label value is assigned to each of the regions constituting the road region. Among the intersections between the dividing lines and the boundaries of the areas, the intersections marked with a circle are label displacement points where the label value changes. First, the image is scanned in the horizontal direction, that is, in the x direction in each of the y coordinate devices in the list obtained in the processing of step 4501. In this scanning, when moving from a background area having a label value of 0 to a road area having a label value other than 0, or when moving from a road area having a label value other than 0 to a background area having a label value of 0, Create a new region boundary point structure.

This region boundary point structure is shown in FIG. As shown in the drawing, a frame is newly provided in the x direction corresponding to each boundary point located at the intersection of the boundary of each area and the dividing line. Each new frame is connected to each frame of the y-direction region boundary point structure created in the above-described steps. This connection relationship is represented by a pointer represented by an arrow shown, and the frames are linked in the horizontal direction by the pointer.

Next, the process of step 4503 will be described with reference to FIG.

It is determined whether or not each set of points in the horizontal direction of the region boundary point structure created in the process of step 4502 exists in the same region. The determination as to whether or not the areas are the same is made based on the feature list obtained in the above-described “method of merging a plurality of areas”. When adjacent points exist in the same area, the boundary point located on the left side is a left edge, and the boundary point located on the right side is a right edge, and this attribute is described in each list. With this attribute, the left and right ends of the area can be distinguished. Further, the distance in the real space from the left edge to the right edge is calculated, and similarly described in each list. In this distance calculation, it is necessary to execute a reverse projection conversion process, which is a process reverse to the above-described projection conversion, to obtain an actual road region from the image.

For the image shown in FIG. 48 (a), a method of identifying edges will be specifically described. Road area 4801 has three areas A,
B and C are combined and expressed. Here, the connection labels of the above-described feature amount list are shown as in FIG. That is, the area C is described in the connection label section of the area A, nothing is described in the connection label section of the area B, and the area A is described in the connection label section of the area C. Therefore, it is determined that the area A and the area C are the same area to be merged, and the area B is the area A and the area B
It is determined that this is an area different from the area formed by.

Therefore, when each boundary point is horizontally image-scanned along the dividing line 4802, the boundary points a, b, c, and d marked with と し て are obtained as label displacement portions. Therefore, the boundary point a and the boundary point b
Exist as points on the same area, the boundary point a is given the attribute of the left edge, and the boundary point b is given the attribute of the right edge, and are described in the list of the area boundary point structure. The boundary point c and the boundary point d are in the same area because their label values are the same, and the boundary point c is a left edge, a boundary point d.
Has a right edge attribute, and is described in the list of region boundary point structures.

Next, the processing in step 4504 will be described with reference to FIGS.
This will be described with reference to FIG.

In the image shown in FIG. 49 (a), each boundary point located at the intersection of the left boundary line and each partition line is denoted by ai, ai + 1,
Each boundary point located at the intersection of the right boundary line and each division line
bi and bi + 1. Here, it is determined whether or not the i-th and (i + 1) -th area points in the y direction of the area boundary point structure exist in the same area. That is, it is determined whether the boundary points ai and ai + 1 are in the same area, and whether the boundary points bi and bi + 1 are in the same area. In the case shown in the figure, since each boundary point is in the same region, in the region boundary point structure shown in FIG. 3B, the boundary points located before and after the screen are linked by a pointer indicated by a thick line. .

If there is an internal point between the left edge and the right edge, the boundary of the area is tracked as shown in FIG. FIG. 13A shows a road area 5001 to be processed.
Is an image in which a white line 5002, which is a lane dividing line, is drawn in the center of the image. The label value of this white line 5002 is 0. Here, when the image is scanned in the horizontal direction along each section line, the internal points E1i and Ei shown in FIG.
+1 and E2i and E2i + 1 are obtained. Therefore, the boundaries of the area are tracked to explore the relationship between each interior point. When this boundary is traced from E1i along the boundary of the white line 5002, an internal point E1i + 1 is reached. When the boundary is traced from the internal point E2i, the internal point E2i
Reach 2i + 1. Thus, the adjacent internal points E1i, E2i
Is the next interior point E1i + 1, E2i on the partition line located above
When the number is consecutive to +1, an attribute is added such that the inner point E1i located on the left side is an inner point left edge, and the inner point E2i located on the right side is an inner point right edge.

When the internal points reach adjacent internal points without continuation of the internal points on the upper dividing line by tracking the boundary, an attribute of a hole is given to each adjacent internal point. That is, the interior point
Following the boundary from E1i + 1, the adjacent interior point E2i + 1
Reach In this case, an attribute called a hole is given to each of the internal points Ei + 1 and E2i + 1. The above process is performed along each y coordinate of the list until the y coordinate reaches the horizon position 5003. At this time, the distance L in the real space from the left edge of the internal point to the right edge of the internal point is calculated. This calculation is obtained by subtracting the x coordinate value of the left edge point in the real space from the x coordinate value of the right edge point in the real space.

Next, the process of step 4505 will be described with reference to FIG.

In the method of assigning attributes by the processing of each step described above, a correct attribute is not assigned when a road branches in a Y-shape or when roads merge. That is, there is a case where the front-back direction association of each boundary point is not performed correctly. Therefore, in such a case, in this step, the boundary is tracked and the evaluation of the boundary is corrected.

For example, it is assumed that the image shown in FIG. The road area 5101 in this image is branched in a Y-shape, and is divided into a main road 5102 and a branch road 5103. Each division line (i, i + 1, i + 2,...) Is image-scanned from the bottom to the top of the screen, and the image is scanned from the left to the right of the screen at each division line. As a result of this scanning, the boundary point located at the intersection between the leftmost boundary line and each of the division lines is denoted by the symbol a, and the boundary point located at the intersection of the boundary line and the division line one position to the right from the leftmost boundary line is located. The boundary point is denoted by b, and the boundary point located at the intersection of the boundary line and the division line located two right to the left end boundary line is denoted by c. As a result, the region boundary point striker shown in FIG.

The association of each boundary point by the area boundary point structure does not match the actual road boundary as understood from the figure, and each boundary point is not given a correct attribute. Therefore, the boundary of the area is tracked from the interior point, and if the next point is a boundary point existing at the left or right edge of the area, it is determined that the interior point and the next point are connected, The link of each point is replaced. In the case of the figure, the boundary of the area is traced from the internal point ci + 4. When tracking, the next point is ai + 5. Since this point is the left edge of the area, the link is replaced. That is, the attribute assigned to each boundary point after ai + 5 is the same attribute as c as the internal point ci + 4. By this link replacement processing, FIG.
Is obtained. This structure conforms to the actual road boundaries.

FIGS. 9D and 9E are diagrams showing the state of the area boundary before and after the link replacement processing by vector expression. That is, FIG. 11D shows a vector state before the link replacement, and the vector represents the connection relationship between the boundaries based on the area boundary point structure shown in FIG. The vector with the attribute a and the vector with the attribute c do not correspond to the actual road boundary. FIG. 11E shows a vector state after the link replacement processing is performed, and the vector represents the connection relationship of each boundary line based on the modified area boundary point structure shown in FIG. . The terminal part of the vector to which the attribute a is attached is modified as described above, and is replaced with the attribute c. For this reason, the boundary of the road area represented by each of the vectors a, b, and c has a shape conforming to a real thing.

(F) and (g) are diagrams expressing the state of the region boundary when the roads merge, by using a vector in the same manner as described above. FIG. 11F illustrates the area boundary before the link replacement by a vector. The vector with the attribute d blocks the entrance of the merging path formed by the vectors with the attributes e and f. FIG. 9G illustrates the area boundaries after the link replacement processing similar to the above by using vectors. The start portion of the vector to which the attribute d is attached is corrected to the attribute f by the link replacement process, and the area boundary matches the actual merging path.

Next, the process of step 4506 will be described with reference to FIG.

For example, assume a road image shown in FIG.
This road image has a road area 5201, and a road shoulder 5203 separated by a white line 5202 is on the left side of the road area 5201. A boundary point is provided at the intersection of the boundary line and the division line of each area of the road image, and a point sequence at the road end and the road shoulder end shown in FIG. Note that points that fall on the edge of the image are excluded from the point sequence. Y obtained by the operation of each of the above steps
The link of each area boundary point structure in the direction is traced upward from a position corresponding to the lower part of the image. When the number of links of the area boundary point structure is equal to or more than a predetermined number, an attribute called a start point of a point sequence is attached to a portion corresponding to the lower part of the image of the area boundary point structure link. The starting point of the illustrated point sequence is indicated by a triangle.

Of the structures with the attribute of starting point,
A set of structures whose distance in the real space from the left edge to the right edge of the region boundary is equal to or greater than the width of the traveling vehicle is defined as a road edge. In the case of FIG. 14B, the dot sequence 5204 and the dot sequence 5205
Is a set of structures corresponding to. Also, there is a set of structures next to the road edge, the distance between this structure group and the road edge is smaller than the width of the structure group representing the road edge, and the distance between the structure group and the other road edge. Is larger than the vehicle width and the actual distance between the structures is continuous at about 20 cm, the structures form a white line. Then, a portion sandwiched between the structure group and one road end is regarded as a road shoulder divided by a white line.

In the point sequence shown in FIG.
A white line is formed by a set of structures equivalent to 07,
A portion between the white line and the road edge 5204 is regarded as a road shoulder.

By the way, the above-mentioned "means for obtaining a travelable range" is for recognizing a traveling course mainly on an area, and is premised on that a road area obtained as a result of image processing is one area. According to this means, it is possible to identify a road end of a monotone road or a branch road. However, when the area is divided by the white line, particularly, when the area is divided into a road and a road shoulder, it is difficult to handle them separately. This is because the means scans the window from the inside of the area to search for the area boundary, so that the road edge and the shoulder edge cannot be distinguished at the same time.

However, as described above, by scanning the label image horizontally from the bottom to the top of the image, tracking the boundary of the local region, and finding the displacement position of the label value of each region, knowing the boundary of the region Can be done. At the same time, by updating the relationship between the regions in the structure and structuring the boundary points of the regions, the road region is structured and represented inside the computer. Therefore, according to the “internal representation method of traveling courses of various shapes”, it is possible to obtain the boundary end coordinates of a plurality of regions such as a road shoulder and a traveling lane, to easily detect the region boundaries, and The target region is not limited to a single region as in “means for obtaining range”. Further, the road shoulder and the like separated by the white line can be identified.

In addition, the required memory capacity is smaller than that of the conventional area boundary method, and the traveling range of the traveling course sufficient for the autonomous vehicle to automatically travel is evaluated by performing sequential boundary search or polygon approximation. It can be obtained without performing such processing. Furthermore, even when the traveling course is not a monotonous road and the road branches or merges,
The shapes of these branch roads and junctions can be expressed in a unified manner, and the boundaries of the traveling course can be accurately identified.

As described above, according to the “internal representation method of traveling courses of various shapes”, according to the attributes described in the list,
The connection relationship between the boundary points of each region is expressed in a structured structure, and the boundary of the target region is identified from the structure. For this reason, since the number of point sequence groups determined as in the related art is large, the process of approximating the boundary of the target area with a polygon and eliminating unnecessary points is eliminated. Further, it is not necessary to sequentially track all the boundary pixels of each region one by one as in the related art, and the target region is identified by processing only predetermined boundary pixels. Therefore, the processing time is shortened, and an area identification method suitable for automatic traveling of the self-contained traveling vehicle is provided.

〔The invention's effect〕

As described above, according to the present invention, the connection relationship between the boundary points of each region is expressed in a structured structure by the attributes described in the list, and the boundary of the target region is identified from this structure. For this reason, since the number of point sequence groups determined as in the related art is large, the process of approximating the boundary of the target area with a polygon and eliminating unnecessary points is eliminated. Further, it is not necessary to sequentially track all the boundary pixels of each region one by one as in the related art, and the target region is identified by processing only predetermined boundary pixels. Therefore, the processing time is shortened, and an area identification method suitable for automatic traveling of the self-contained traveling vehicle is provided.

[Brief description of the drawings]

FIG. 1 is a flowchart showing an outline of a structuring process of a road area in a "method of internally expressing a traveling course having various shapes" according to an embodiment of the present invention. FIG. 2 is a color image processing for extracting a traveling road area. A flowchart showing the schematic flow of
FIG. 3 is a block diagram showing an algorithm of the ISH conversion processing in the "ISH conversion processing of a color image using a ROM table". FIGS. 4 and 5, FIG. 6, and FIG. FIG. 8 is a diagram for explaining the example of the processing in comparison with the conventional example of processing, and FIG. FIG. 9 is a diagram for explaining the state of peaks and valleys in this pre-processing, FIG. 10 is a diagram for explaining changes in table values in this pre-processing, and FIG. FIG. 12 is a flowchart showing an outline of the processing, and FIG. 12 is a flowchart showing details of a traveling path extraction process in “traveling course extraction means by repeated threshold processing using a template image”; Fig. 14 is a diagram for explaining each process in the road candidate region extraction process. Fig. 14 is a process for various input images in "means for extracting a shadow or a high-brightness portion from a traveling course focusing on a difference in brightness". FIG. 15 is a graph showing an example of a brightness histogram for explaining shadows and high-luminance portions in this means, FIG. 16 is a diagram showing a labeling board configuration of a `` labeling processing device '', FIG. 17 is a general flowchart of the labeling process, FIG. 18 is a diagram for explaining a multi-valued input labeling system, FIG. 19 is a diagram for explaining a labeling system using a run, and FIG. 20 is a diagram of the labeling process. FIG. 21 is a view showing a window used in image scanning, FIG. 21 is a block diagram of a labeling processor KLP, and FIGS. 22, 23, 24, 25, 26, and 27; Is a flowchart showing window processing performed in image scanning of the labeling processing, FIG. 28 is a configuration diagram of the labeling memory KLM, FIG. 29 is a view for explaining processing for matching finally applied label values, FIG. FIG. 30 is a configuration diagram of the feature extraction processor KLC, FIG. 31 is a diagram for explaining the one-scan labeling method, and FIG. 32 is an inverse diagram used for searching for a label region in the “unit for merging a plurality of regions”. FIG. 33 is a view showing an L-shaped mask, FIG. 33 is a view showing an example of a divided area used for explaining the area boundary search using the inverted L-shaped mask shown in FIG. 32, and FIG. FIG. 35 is a diagram showing the label-to-label boundary length matrix where the label-to-label boundary length and the perimeter of the region obtained by the character mask scanning method are stored. Feature value to be described Figure showing the strike
FIG. 36 is an example of a label area having each feature described in the feature list shown in FIG. 35, and FIG. 37 is a reference pixel position No. assigned near the pixel of interest in the boundary search method. FIG. 38 is a diagram showing a pixel reference table used in the boundary search method, and FIG. 39 is a pixel area used in explaining how to use the pixel reference table shown in FIG. 38. FIG. 40 is a flowchart showing the flow of the processing of the traveling course recognition system in “means for obtaining a travelable range”, and FIG. 41 is movement of a window in a captured image used in the description of this means. FIG. 42 is a diagram for explaining the values of the window and the histogram in the target area. FIG. 44 is a diagram for explaining, and FIG. 44 is a diagram for explaining a method of correcting the relation of the point sequence group obtained in order to prevent the reversal of the recognition of the road edge shown in FIG. 43 using the Mahalanobis general distance. FIG. 45 is a block diagram showing a schematic configuration of the entire color image processing apparatus, and FIG. 46 is a view showing y in the “internal representation method of traveling course of various shapes”.
FIG. 47 is a diagram for explaining the direction list and the region boundary point structure, FIG. 47 is a diagram for explaining the label displacement portion and the region boundary point structure, FIG. 48 is a diagram for explaining the attribute of the region boundary, FIG. 49 is a diagram for explaining the link processing in the front-back direction of the region boundary point, FIG. 50 is a diagram for explaining the attributes of the internal point edge and the hole, and FIG. 51 is a process of replacing the attribute once given FIG. 52 is a diagram for explaining detection of a road edge and a road shoulder edge. 101: color camera, 102: ISH conversion unit, 103: color processing unit, 104: labeling hardware, 105: CPU processing unit, 148: label image, 149: feature amount list 1, 150
... road position coordinates, 151 ... point sequence image, 4603 ... road area boundary line, 4604 ... partition line, 4605 ... area boundary point structure in y direction, 4606 ... list.

Claims (3)

(57) [Claims] An image labeled in each area obtained by dividing an image is divided by a dividing line in a certain direction, and a list in a direction orthogonal to the certain direction is created for each of the dividing lines. By scanning the image, a change point of the label value of each area is identified to obtain a boundary point located at the intersection of the division line and the boundary line of each area, and a list in the fixed direction is obtained for each of the boundary points. The attribute of the boundary point along the dividing line in the certain direction is obtained based on the connection relation of the label value of each area, and the attribute is described in the list of the certain direction. The attribute of the boundary point along the boundary line of the divided area in the direction orthogonal to the predetermined direction is obtained based on the connection relationship, and the attribute is described in the list in the predetermined direction, and the attribute is described in each of the lists. Based on the attributes The list length of the list in the direction orthogonal to the certain direction is determined, and the point sequence of the boundary points corresponding to the list having the length equal to or more than the predetermined length is determined to correspond to the boundary of the target area. A method for identifying a target area in image processing, characterized by identifying a boundary of the area. 2. When there is an internal point between the boundary edges of the target area, the boundary of the divided area is tracked from the internal point along the direction orthogonal to the predetermined direction, and the next boundary point appears. An internal point, wherein, when the another internal point is connected to another internal point adjacent on the dividing line, another area is formed in the target area by each of the internal points. 2. The method according to claim 1, wherein the target area is recognized. 3. When an internal point exists between the boundary edges of the target area, the boundary of the divided area is traced from the internal point along the direction orthogonal to the predetermined direction, and the next boundary point appears as the target area. If the boundary point is located at the boundary end of the boundary point, it is determined that the interior point and the boundary point are connected, and the connection point between the boundary point and the boundary point obtained before the boundary point is obtained. 3. The method according to claim 1, wherein the boundary of the target region is corrected by cutting the boundary and connecting the boundary point and the interior point.