JPH0410079A - Device and method for processing color image - Google Patents

Abstract

PURPOSE:To exactly extract shadow or discolored part and to enable the high- speed processing of an area boundary part by extracting an objective area based on lightness and saturation information and calculating the boundary end of the area. CONSTITUTION:The color image processor is provided with an image pickup device 101 and an ISH conversion part 107 converting fetched RGB information to the ISH information of lightness, saturation and hue a color processing pat 103 extracting the objective area based on the lightness and saturation, and a labelling part 104 attaching labels to respective areas constituting the extracted objective area. Further, merging means 127-130 executing merge processing between the respective areas after calculating the connecting relation of the respective labelled areas, and area boundary identifying means 132 and 133 calculating the boundary end of the merged area are provided. Thus, the objective area can be exactly extracted from the color image at high speed.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to a color image processing device and a processing method thereof, which are applied to technology for determining travel routes required for autonomous driving of automobiles.

[Conventional technology]

Conventionally, most of the methods for extracting a driving path region are based on edge detection using monochrome images, and techniques for performing this type of driving path discrimination based on color images are rarely provided.

In addition, when handling digitalized color image information, RGB data (three primary color information - R red, G; green, B
; blue), brightness I (degree of brightness of color), saturation S
(degree of vividness of color).

Hue H (value indicating classification regarding color type) etc. are often used as a feature amount. These brightness! , saturation S, and hue H are obtained by substituting RGB data into a predetermined conversion formula. Conventionally, this conversion process from RGB data to ISH data has been performed every time ISH data is required in image processing. In other words, when ISH data is needed, the CPU
2 image memories storing data, 6 image memories storing G data, and 8 image memories storing B data are accessed to collect RGB data. And C
The PU calculates the necessary ISH data by performing a predetermined conversion process based on this RGB data.

Furthermore, in the conventional method of determining the driving path area, white lines and landmarks drawn on the driving path are detected by edge detection using a monochrome image, and are performed based on these white lines and landmarks.

Furthermore, when an autonomous vehicle runs on a road in a general environment rather than on a limited driving course, various road surface conditions are possible. In particular, when various three-dimensional objects exist around the driving course, shadows are formed on the road surface. If there is a shadow on the road surface, the running road area will not have a negative color.

Therefore, in order to extract the driving road area by image processing,
This must be done taking into account the shadows in the area of the driving path.

Conventionally, in order to recognize this shadow, some research has focused on limiting the road surface conditions of the driving course, measuring the color distribution of sunny areas and shadow areas in advance during these limited times, and then extracting the driving road area. It is carried out in This method saves the measured color distribution of each part on a computer, and when the actual image is imported, reads out the saved color distribution and compares it with the color distribution of the input image. This is to determine the road area.

In addition, in order to determine the travel path area, it is necessary to label each area into which the image has been divided and to classify each area. This labeling process is essential for calculating the area and center of gravity of a target region within a binary image. Labeling algorithms that have been proposed so far include:
It can be roughly divided into Rusk scanning type, Rancourt type, and boundary tracking type. However, when converting to hardware (speeding up), the rask scanning type is generally used because of its real-time processing potential, circuit scale, and ease of implementation. This scan first allocates temporary labels in the first scan and retains information indicating connections between temporary labels with different values.
Then, by analyzing the integrated information, a final label that corresponds one-to-one with the target area is obtained. Finally, labeling is performed by scanning the temporary label image obtained in the first scan again and replacing the temporary label with the final label.

Furthermore, when a given image is divided by the area division method, the image area is over-divided. Therefore, based on the color of each over-segmented region and the degree of connection of each region with surrounding regions,
Perform merging processing of each over-divided area. That is, after labeling each divided area, the connection relationship between each label is described.

Then, based on this description, a label change operation is performed to rewrite the label of the area to be merged with the label of the area to be merged. Relabeling is performed by scanning the image and changing the label value, that is, the pixel value, if the image is a pixel in an area to be merged.

Furthermore, conventionally, when determining a road edge or a driving course from an image segmentation region, the boundary between the obtained driving course region and the background region is traced to obtain a point sequence corresponding to the boundary line.

In other words, the pixels located at the boundary of the driving area are tracked one by one along this boundary, the traces of this tracking are made into a series of points, and the edge of the road is determined based on this series of points.

[Problem to be solved by the invention]

Performing this kind of image processing using color images is
The cost is higher than processing using monochrome images, and there are few examples of color image processing in the industrial field.

Further, the conventional conversion process from RGB data to IHS data was performed every time IHS data was required in image processing, and therefore the process required time. In particular, the conversion process from RGB data to hue data H takes a very long time.
Even with n3), this conversion process required about 1 minute.

For this reason, in image processing for autonomous driving of automobiles, etc., which requires real-time processing, calculation processing takes time, and sufficient driving control cannot be performed.

In addition, the conventional method using edge detection in monochrome images described above recognizes the driving course by tracking white lines and landmarks drawn on the driving road, so white lines and landmarks are always required on the driving road. . In addition, as a result of processing images captured by a camera, the detected edges may be interrupted due to the condition of the light source illuminating the driving path or unclear white lines. The area is difficult to detect.

Furthermore, if there is dust or shadows on the road, white lines or landmarks may be falsely detected, and accurate road information may not be obtained. Further, when the boundary between the driving path region and the background region is divided by grass or soil, the change in the edge corresponding to the boundary portion is small, making it difficult to recognize the boundary. In addition, if the brightness of the driving path changes from moment to moment depending on the weather or the position of the sun, the correct driving path area can be determined by fixing the threshold value for dividing the driving path area and the background area. I can't do anything.

Further, as described above, on a general road, the brightness distribution on the road surface changes from moment to moment depending on the weather at the time of driving, the position of the sun, movement of clouds, etc.

For this reason, with the conventional method of measuring the color distribution in advance and comparing it with the input image, this comparison and judgment process cannot follow the changes in brightness mentioned above, and shadow areas cannot always be extracted. It's not necessarily possible. Further, if a part of the road is discolored due to paving work or the like, the road area of the discolored part may not be extracted by image processing depending on the degree of discoloration.

An object of the present invention is to provide an image processing method that can accurately extract shadows and discolored parts even on ordinary roads.

Furthermore, all images targeted for labeling processing up to now have been binary images, and it has been impossible to label multivalued images. Further, in the rask scanning type, it is not possible to attach a label that corresponds one-to-one to the target area in one scan. Further, although primary labeling and secondary labeling can be implemented relatively easily in hardware, analysis of integrated information, that is, integration processing, is difficult to implement in hardware. In addition, in the rask scanning algorithm, provisional labeling sometimes generates a large amount of integrated information, so the integration processing load is large, and the processing performance of the labeling process as a whole does not improve.

Furthermore, the conventional label value replacement operation for merging regions is performed by scanning all pixels of an image, or at least pixels within a rectangle circumscribing the merging region. Therefore, labeling requires time for replacement scanning, making it difficult to perform high-speed processing required for autonomous driving of automobiles. Another disadvantage is that processing costs are not reduced.

Furthermore, the conventional method of determining the road edge by sequentially tracking each pixel at the region boundary takes time. Furthermore, when there are multiple areas, it becomes a problem as to what criteria should be used to determine the edge of the road. Furthermore, the driving course area and the divided areas are connected very slightly, and depending on the result of image processing, it is possible that this divided area may not be the driving course. In such a case, with the conventional method of sequentially tracing the boundaries of regions, boundaries of regions that are not actually the driving course are also found as a series of points.

As a result, accurate road edge information cannot be obtained. Further, even if the obtained point sequence groups are separated without being connected, it is necessary to verify the validity of a plurality of point sequence groups, which complicates the processing. Furthermore, if the obtained image has holes or cuts due to noise, the obtained point sequence will not be clear.

[Means to solve the problem]

The present invention has been made to solve these problems, and includes an imaging device that captures an original image as RGB information, and an ISH that converts the RGB information captured by the imaging device into ISH information with one hue of brightness and saturation. A conversion unit, a color processing unit that extracts a target area based on brightness and saturation from the ISH information converted by the ISH conversion unit, and a labeling unit that labels each area constituting the target area extracted by the color processing unit. a labeling unit that performs a merging process between the areas by determining the connection relationship between the areas labeled by the labeling unit, and an area boundary identification unit that determines the boundary edge of the merged areas. It is something to be admired.

Further, an original image memory stores original image data, an R image memory stores R data of RGB data included in the original image data, a 0 image memory stores G data, and a 0 image memory stores B data. 8 image memories, a first storage element that stores brightness data converted from RGB data, a second storage element that stores chroma data converted from RGB data, and a second storage element that stores saturation data converted from RGB data. This device includes a third storage element in which hue data is stored, and an average value filter that averages the hue data output from the third storage element.

In addition, RGB data is extracted from the captured original image data, this RGB data is converted into brightness data and saturation data, a feature histogram is created from each data, and a region segmentation threshold is determined based on this histogram. By setting and binarizing the original image using this threshold, the area is divided into divided images, and the image is binarized into a predetermined area assumed at the image position where the driving road is imaged and other areas. The system calculates the degree of overlap between the template image and the divided images, determines that the divided image with a high degree of overlap includes the running road area, and extracts the running road area.

In addition, if there is an area darker or brighter than the driving road area extracted from the brightness data, this dark area or bright area is binarized based on the valley candidate value obtained based on the shape of the feature value histogram, and A dark area or a bright area is determined based on a binary image and a travel road area image determined from a chroma image.

In addition, there is an input memory that stores multivalued image information obtained by adding up multiple binary input images, a labeling processor that labels each pixel stored in the input memory according to the pixel value, and a labeling processor that labels each pixel stored in the input memory according to the pixel value. and a label memory for storing the output label value of each pixel. Furthermore, a labeling memory is used as the label matching memory and the label memory in which information on each element can be rewritten simultaneously.

Additionally, a line buffer memory is used to label each run of the image.

In addition, by scanning the mask for the pixels of each area that has been segmented and labeled, the perimeter of the same label area and the common boundary length between the areas adjacent to the same label area are calculated, and the perimeter and common boundary are calculated. The degree of connectivity between adjacent regions is determined based on the ratio to the length, and this degree of connectivity is written in memory to determine the connection relationship between each region.

In addition, a pixel reference table is created in advance in which each pixel located near the pixel of interest is given a number indicating its position, and the position numbers are arranged in a predetermined order in the column and row directions. Find the perimeter of the region and the common boundary length between adjacent regions by searching for pixels near the pixels on the boundary of , this degree of connectivity is written in memory to determine the connection relationship between each area.

In addition, a window surrounding a predetermined number of pixels is scanned in the horizontal direction of the image-divided target area, and the number of pixels with a predetermined pixel value existing within the window is counted for each scanning position, and the number of pixels with a predetermined pixel value is calculated. When there are consecutive window positions where the number of pixels is less than or equal to a predetermined number, the window position where the number of pixels starts to become less than or equal to the predetermined number is determined to be the boundary of the target area, and the boundary of the target area is identified.

[Operation] The conversion process from RGB data to ISH data is performed only when storing ISH data in each storage element, and thereafter the ISH data can be directly read from each storage element.

Furthermore, the relationship between each image divided into regions based on the threshold value and the target region is determined by calculating the degree of overlap with the template image.

In addition, similar color areas with lower brightness than the target area are extracted individually as corresponding to parts such as shadows formed in the target area, and similar color areas with higher brightness than the target area correspond to parts such as sunlight shining on the target area. They are extracted individually as corresponding parts.

The labeling processor also labels multivalued input images. Furthermore, the labeling memory can simultaneously rewrite the information stored in each register, reducing the labeling processing time and reducing the size of the concave path.

In addition, labeling is performed by determining the merging relationship between the divided regions without performing scanning.

Further, the range of the target area is determined based on the number of pixels existing within the window. In other words, the boundary between the target area and the background area is identified by scanning the window horizontally from side to side over the target area and examining changes in the number of pixels within the window.

〔Example〕

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A case where a color image processing apparatus and a processing method thereof according to an embodiment of the present invention are applied to driving control of an autonomous vehicle will be described below. With the device and method of this embodiment, the driving course of the vehicle is automatically recognized, and the vehicle determines the steering angle, fuel injection amount to the engine, etc. based on the recognized driving course, and autonomously runs the vehicle. do.

FIG. 1 is a block diagram showing a schematic configuration of the entire color image processing apparatus according to this embodiment.

The color image processing device includes a color camera 101 that captures road information, and an I that performs SH conversion on the captured RGB information.
An SH conversion unit 102, a color processing unit 103 that extracts road candidate areas etc. from the ISH converted image information, a labeling hardware unit 104 that performs labeling processing on the color-processed image, and a merging process on the labeled image areas. It is roughly divided into a CPU processing unit 105 that executes the following operations. The ISH converting unit 102 includes a color camera input board, an ISH converting board, a filter, and the like.

FIG. 2 is a schematic flowchart showing an image processing method in this color image processing apparatus.

A road image 106 is captured by the color camera 101 as RGB information, and the ISH conversion unit 102 converts the brightness (
1), each image is converted into one image of saturation (S) and hue (H) (
Step 201). Based on each of these images, the color processing unit 103 extracts a road candidate area that will become the basis of a driving course (step 202). Here, it is determined whether the low brightness threshold existence flag in the status register of the CPU is on (step 203). This flag indicates whether or not there is a low brightness area such as a shadow or a discolored part in the captured original image. If the flag is on, the low brightness area is extracted in the color processing unit 103 ( Step 204). The extracted road candidate areas and low-luminance areas are labeled by the labeling hardware unit 104, and the connection relationship between each area is determined.

Based on this judgment result, if each area is in a relationship that should be merged, the CPU processing unit 105 (\merge)
Processing is executed (step 205).

Next, it is determined whether the high brightness threshold presence flag in the CPU status register is on (step 206). If there is no step 203 (even if the low-luminance threshold presence flag is turned on), the process of step 206 is immediately executed. This high brightness threshold presence flag is a flag that indicates whether or not there is a high brightness area such as sunlight or discolored areas in the captured original image.If the flag is on, the color processing unit 103 (The high-brightness area is extracted with the cover (Step 207).

The extracted road candidate areas and high brightness areas are labeled by the labeling hardware unit 104, and the connection relationship between each area is determined. Based on the result of this determination, if the areas are in a relationship that should be merged, the CPU processing unit 105 executes a merge process (step 208).
).

Based on the road candidate areas merged in this way, the left and right boundary edges of the area, that is, the boundary lines of the road edges, are determined as a dot sequence. Using this point sequence information, the driving course based on the original image taken this time is recognized (step 209).
. Thereafter, the process returns to step 201, and the same process as described above is repeatedly performed on image information that is subsequently obtained as the autonomous vehicle moves, and travel control of the autonomous vehicle is executed.

Next, the above processing will be explained in more detail based on the configuration diagram of the color processing apparatus shown in FIG.

A color camera 101 is fixedly installed on the body of an autonomous vehicle, and the color camera 101 captures a road image 106 located in front of the vehicle as RGB information. The conversion processing unit 107 of the ISH conversion unit 102 has this R
GB information is given. In this conversion processing unit 107, a color image ISH conversion process using a FROM table, which will be described in detail later, is executed, and the RGB road image information is converted into a brightness (1) image 108. It is converted into image information of a saturation (S) image 109 and a hue (H) image 110.

The brightness image 108 is given to the road candidate area extraction means 111, and a road candidate area image 113 based on the brightness image is extracted by a "driving course extraction method by iterative threshold processing using a template image" which will be described in detail later. . Further, the chroma image 109 is provided to the road candidate region extracting means 112, and a road candidate region 114 based on the chroma image is extracted using the same method as described above. The threshold value for region segmentation in this method is determined by "threshold setting means based on the shape of the feature histogram in iterative threshold processing" which will be described in detail later. The obtained road candidate area images 1, 13, 114 are given to the logical product calculation means 115, the common part of each road candidate area obtained from the brightness and saturation is extracted, and a new road candidate is created based on the color information. This becomes a region image 116.

Furthermore, if a low brightness area exists in the brightness image 108, a low brightness threshold presence flag in the status register 117 of the color processing section is turned on. When this flag is on, the low brightness area extraction means 118 captures the brightness image 108. The low-luminance area is then extracted using a ``extraction means for extracting shadows and high-luminance areas from the driving course that focuses on differences in brightness'', which will be described in detail later. The extracted low brightness area becomes a low brightness image 119. This low brightness area image 119 is logically ANDed with the road candidate area image 114 extracted from the saturation image 109 in the logical product calculation means 120, and the low brightness area has a color saturation similar to that of the road candidate area. A region is extracted. The extracted image information of this low brightness area is added to the road candidate area image 116.

Furthermore, if a high brightness area exists in the brightness image 108, the high brightness threshold presence flag in the status register 117 of the color processing section is turned on. When this flag is on, the high brightness area extraction means 121 captures the brightness image 108. Then, the high-brightness area is extracted using a ``extraction means for extracting shadows and high-brightness areas from the driving course that focuses on differences in brightness'', which will be described in detail later. The extracted high brightness area becomes a high brightness area image 122. This high brightness area image 122 is logically ANDed with the road candidate area image 114 extracted from the saturation image 109 in the logical product calculation means 123, and the high brightness area has a saturation similar to that of the road candidate area. A region is extracted.

The extracted image information of this high brightness area is added to the road candidate area image 116.

By each of the above means, the road candidate area image 116 includes a road candidate area image 113 with a pixel value of 1, a low brightness area image 119 with a pixel value of 2, and a high brightness area image 122 with a pixel value of 3. Become.

The road candidate area image 116 is provided to the labeling processing section 124. The labeling processing unit 124 labels each region of the given image and creates a label image 126. At the same time, the labeling processing unit 124 calculates the area, center of gravity, etc. of the same label area. Each of these calculated values is calculated as a calculated value 125 corresponding to the labeling processing unit 124.
become. This labeling process is executed by a "labeling processing device" which will be described in detail later.

The CPU takes the label image 126 into the small area removing means 127. Here, it is assumed that small areas caused by noise or the like and areas whose center of gravity is located above the horizon position do not correspond to road areas, and these areas are removed from each label image. The label image from which the small area has been removed from the road candidate area by the small area removing means 127 becomes a new label image 128 . In addition, among the calculated values 125 for each label attached to each region, the features of each region not removed by the small region removing means 127 are recorded in the feature list 1.

Feature amount 129 stored in list 1. Label image 128
．． Based on the road image 116, each label area of the road candidate area image and each label area of the low brightness area image, and each label area of the road candidate area image and each label area of the high brightness area image are in a relationship that should be merged. In this case, the area merging means 130 executes the merging process, and a new label image 131 is stored in the memory 1. The above merging process is executed by a "multiple area merging means" which will be described in detail later.

Road position coordinates 132 of the left and right ends of the road area are calculated based on the label image 131 finally obtained. Based on the road edge position coordinates 132, point sequence data 133 corresponding to the road edge is determined. This process of determining the road edge is executed by a ``means J for determining a drivable range'' and an ``internal representation method for driving courses of various shapes'' which will be described in detail later.

This point sequence data 133 is sent to a data management unit that performs overall management of image processing data for driving control of the autonomous vehicle, and color image processing is completed.

Next, ISH of a color image using a FROM table that converts RGB data obtained by capturing an image with a color camera to brightness I and saturation 8
The "conversion process" will be explained below using FIG.

First, an original image 301 is input as RGB data from a color camera. Since the original image 301 contains a mixture of RGB data, each component of the RGB data is separated. Then, each separated RGB data is stored in an R image memory 302,
The images are stored separately in each of three image memories, 0 image memory 303 and 8 image memory 304. Each of these R, G,
The B image memories 302 to 304 are composed of a plurality of pixel values having 8-bit gradation, and are subjected to ISH conversion as follows.

First, when reading each 8-bit RGB data, only the upper 6 bits are read and the lower 2 bits are discarded.

That is, by taking the upper 6 bits, a pixel value of 8 gradations is approximated to a pixel value of 6 gradations. The value of the upper 6 bits changes between 00 and 3F (hex) in hexadecimal (00 to FC when the lower 2 bits are rounded down).
(hex)). In addition, the value obtained by dividing this upper 6 bit value by 40 (hex) is calculated as R and G, respectively.
.

Let it be the B value. Each of these R, G, and B digits varies in the range of real numbers from 0 to about 1.

Assuming that the values obtained by dividing each of R, G, and B by (R+G+B) are r, g, and b, respectively, the conversion from R, G, and B values to I, S, and H values is performed according to the following equation.

Here, min (r, g, b) indicates the value of the feature amount having the minimum value among the feature amounts r, g, and b.

1 - (R+G+B)/3
... (1) S=1 - (1/3) ・m
in (r, g, b)H=1/2+(1/π
) ・arcta'n ( (3) "2 (g-b)
/ (2 r-g-b) 1 This conversion is performed for each R, G, and B pixel in the order of rask scan, and the converted 1.3 and H values are all real numbers that take values from 0 to 1. becomes. ROM 305 has conversion table data from RGB to brightness I, ROM 306 has conversion table data from RGB to saturation S, and ROM 307 has conversion table data from RGB to hue H.
Conversion table data is stored.

Each of these ROM'305-307 functions as a lookup table. Further, the address at which data is stored in each ROM 305 to 307 is determined by the upper 6 bits of the 8 bits of each R, G, B value before conversion. In addition, each ROM 305 to 307 has 8 memory trays.
It is 8 bits (-256 bytes).

The features stored in the ROMs 305 and 306 are immediately read out as brightness data 308 and chroma data 309 in response to a fetch command from the CPU, and are supplied to necessary image processing in real time. Furthermore, the characteristics of the hue data 310 read from the ROM 307 are as follows:
Noise is removed and averaged by a 3×3 average value filter 311. Therefore, the hue data 312 used for each image process is smoothed data without noise. Further, each of the read feature data (brightness 308, saturation 3091, hue 310 and 312) is 8-bit approximate data in which the lower 2 bits are 0 and the upper 6 bits are valid.

Next, an example of ISH conversion processing using this algorithm will be described with reference to FIGS. 4 to 7, while comparing it with an example of ISH conversion processing that does not use this algorithm.

(a) of each figure is an outline of an original image captured by a color camera. In other words, FIG. 4(a) shows a scene in which a running road curves in the distance, a guardrail is installed on one side of the running road, and trees are growing in the distance from this guardrail. FIG. 5(a) shows a road where the edges of the road are divided by weeds, etc., and there are houses, trees, etc. in the distance. FIG. 6(a) shows a driving route on an expressway at night, and the road surface is faintly illuminated by moonlight and lamps. 7th
Figure (a) shows a driving route on a sunny day, and the driving path is shaded by trees inside a block wall.

In addition, (b-1) and (b-1) of each figure from FIG. 4 to FIG.
-2) is a histogram with brightness I as the feature, and (
c-1) and (c-2) are histograms with saturation S as the feature, and (d-1) and (d-2) in each figure are 3×3
Histograms with hue H' as a feature before being applied to the average value filter 311, (e-1) and (e-2) in each figure
is a histogram whose feature is the hue H after being applied to the 3×3 average value filter 311.

The vertical axis of each histogram indicates the number of pixels between each feature, and the maximum is 1/4 of the entire screen. Further, the horizontal axis of each histogram indicates the degree of each feature of brightness I and saturation 81 and hue H'H. The degrees between these features are displayed so that they become stronger as they move away from the origin, and range from 0 to FF (h e x
) The degree between each feature assigned to each numerical value is divided into 64 and displayed. Also, the number of pixels of the original image is 512X
It is slightly smaller than 512. This is because there is a portion in the periphery of the image where all R, G, and B data are 0, and each histogram is represented by data values that include this portion.

Also, (b-1), (c-1), (d
-1) and (e-1) are histograms obtained based on the conventional method, in which the CPU directly accesses each image memory and converts R, G, and B data into I, S, and H data according to the conversion formula. This was obtained by In contrast, (b-2) in each figure.

(c-2), (d-2), and (e-2) are histograms obtained based on the algorithm according to the method of this embodiment. This is obtained by storing and reading out a conversion table between each feature.

Each (b-1), (b-2) and each (
As shown in c-1) and (c-2), it can be seen that there is not much difference in brightness I and saturation S between the histogram distribution according to the present method and the histogram distribution according to the conventional method. This means that the function of this method to convert RGB data to ISH data is different from that of the conventional method.
This shows that there is no inferiority in terms of functionality. on the other hand,
The histogram distribution of the hue H' between the features before being applied to the 3X3 average value filter 311 shown in (d-1) and (d-2) of each figure shows an overall tendency between the conventional method and the present method. It has changed. In the image used in this example, this is the result of (g-b
) and (2r-g-b> take only extremely limited values around O, this is because the information is extremely discretized by compressing data to 6 bits using this method.

However, as shown in (e-1) and (e-2) of each figure, the histogram distribution obtained by applying this hue H' to the 3×3 average value filter 311 is not sufficient to the distribution obtained by the conventional method. It is compatible. The calculated value of the hue data is an unstable value that easily changes due to small noise in the RGB data, and the hue image has very large noise. For this reason, as in this method,
Understand that it is appropriate to perform some kind of smoothing on the calculated value results of hue conversion, and that the same problem will not occur even if the feature values are compressed to 6 bits and processed by performing this smoothing. be done. Note that the calculated value of hue is an unstable value that easily changes due to noise, so the hue H′
By passing the data through an average value filter, a distribution similar to the histogram distribution obtained by the conventional method was obtained.

In this way, by using the ROMs 305 to 307 as look-up tables and converting each of the I, S, and H feature quantities in advance, the CPU is not required to store each image memory each time processing is required, unlike in the past. There is no need to access and calculate. As a result, the arithmetic processing speed during data conversion according to this embodiment can be executed at high speed at the video rate, and the processing speed is improved. In addition, the above-mentioned R, G,
Even if the conversion formula from B data to I, S, H data changes, it can be handled by the same hardware. In other words, the effect of changing this conversion formula is
There is only a change in the memory contents of 305 to 307. Therefore, the hardware does not change even if the conversion formula is changed.

Furthermore, since each pixel value of R, G, and B is compressed to 6 bits, the amount of hardware can be reduced. Furthermore, by passing the hue H through the 3×3 average value filter 311 when reading out the hue H from the lookup table, it is possible to prevent adverse effects caused by compressing the data to 6 bits, such as discretization of the histogram.

Next, "threshold setting means based on the shape of the feature amount histogram in repeated threshold processing" will be explained.

This means is performed as a pre-processing of color image processing for extracting the travel road area.

FIG. 8 is a flowchart showing an outline of this processing process, and shows the processing when the contrast of the original image data obtained from a color camera installed in a running vehicle is low.

First, a histogram representing the frequency with respect to color feature quantity (brightness or saturation) is created based on RGB digital image data of an original image obtained from a camera. The horizontal axis of this histogram is set to the color feature amount (lightness or saturation), and the vertical axis is set to the frequency of the feature amount.

This histogram has low contrast in the original image, so
The distribution is biased towards the origin of the histogram.

Furthermore, images with low contrast tend to have a single peak mode, and no clear valleys occur. Therefore, a histogram is created after image enhancement is performed by performing predetermined calculations on each pixel value in the film, but in the case of this method, this histogram is displayed on the histogram data in the Image enhancement is performed by simply stretching the image (step 801).

Then, left end processing (step 802) and right end processing (step 803) of the emphasized histogram are executed. Next, from the frequency distribution state of the histogram, a temporary top (peak) with a high frequency for the feature amount and a temporary trough with a low frequency for the feature amount are set on the peak/trough table (step 804). Evaluation of valleys is performed on this table based on the frequency of temporary valleys obtained (step 8).
05). Furthermore, the valley is evaluated again on the table based on the peaks adjacent to this valley (step 806). Finally, smooth the evaluated peak/valley table,
From the relative relationship between peaks and valleys, valleys that are effective for segmenting the target area of area segmentation and this background area are extracted (step 807
).

On the other hand, in parallel with the above processing, Otsu's discriminant analysis method is applied to the histogram created in step 801, and area segmentation thresholds based on this discriminant analysis method are obtained. and,
The obtained threshold value is compared with the position of the valley extracted in step 807, the frequency of valleys located near the threshold value is taken as the true threshold value, and the threshold value based on Otsu's discriminant analysis method is determined. to correct.

Next, details of each process will be explained below.

Setting of a temporary peak and a temporary valley in step 804 is performed as follows. That is, tentative peaks and valleys are determined based on the distribution of frequencies for the feature amounts. Specifically, depending on the frequency of adjacent points on the left and right of the point of interest on the histogram, the states of peaks and valleys are determined as shown in Figures 9(a) to (f).
) is divided into six states shown in the figure. Here, the point of interest on the histogram is hl (the subscript i means any one point when N points are equally distributed along the horizontal axis of the histogram, and the value is from 0 to N-1). ), focus point h
The adjacent point with a feature value smaller than 1 is h j−1, the focus point h1
Let h Ill be an adjacent point with a larger feature amount. Also,
The difference in frequency between the point of interest h+ and the adjacent point hj-1 is expressed as the pitch pif (pil-hl-hi-1), and the adjacent point h
141 and the point of interest h1 as pitch p12
(p I2- h i +1- h i ).

Range from i-1 to N-2 (excluding both end points of the histogram)
In this case, the peak/trough table value pk1+ is set as follows. That is, when the sign of pif and the sign of pH2 are different, (1) If pj1≧0,゛ and pI2≦0, the table value pkti −1 (2) If pH≦0 and p12≧0, the table value Value pktl--1 FIG. 9(a) shows a state where pH>0 and p I2-0, and therefore the table value pkt+-1. Same figure (
b) is a state where p il > 0 and p 12 < O, so the table value pkt! -1. In the same figure (c), the state is pit-0 and p I2 < O, and therefore the table value is pkti-1. The same figure (d
) is in a state where pll < 0 and pH 2-0,
Therefore, the table value pkti=-1. In the same figure (e), there is a state where p il < 0 and p12 > 0,
Therefore, the table value is pktj--1. FIG. 5(f) shows a state where pil-0 and p12>0, and therefore the table value is pkt1--1.

In this way, from the relationship of relative frequencies of adjacent points, the states of (a), (b), and (c) in the same figure are pkti=1
, and the point of interest hi is determined to be a temporary peak. Figures (d) and (e).

The state of (f) is calculated to be pkti--1, and the point of interest hi is determined to be a temporary valley.

Furthermore, the process of evaluating valleys based on frequency in step 805 is performed as follows. In other words, when the point of interest h1 is a temporary peak in i-1 to N-2 (pkti=1
), (1) The ratio between the adjacent point hj and the point of interest h1 is smaller than 0.1 (hj /hi <0.1) The adjacent point hj is the point of interest hi
, the table value pktj corresponding to the adjacent point hj is set to -1 (pktj=-1).

(2) If the ratio between the adjacent point hI11 and the point of interest hi is smaller than 0.1 (hm /h+ <0.1) and the adjacent point hm is on the right side of the point of interest h1, then the table value corresponding to the adjacent point hlO Set pktIIl to -1 (pktIll -
-1).

Furthermore, the valley evaluation process based on adjacent peaks in step 806 is performed as follows.

In i-0 to N-1, (1) When the point of interest hi is a temporary valley (pktl −-1), ■ A temporary peak on the left side of the valley (adjacent point hk, table value pk
tk-1) will be called topL.

■Temporary peak on the right side of the valley (adjacent point hj, table value pk
tj-1) will be called topR.

(2) When both topL and topR exist at the point of interest h+, ■Ratio g between the point of interest h+ and topL (ρ-hl/1o
pL) is the ratio r(r-hj/1
opR) is smaller than (Ω<r), the table value pk
Add 1 to tj.

■The ratio ρ (hj/1opL) is the ratio r (hi/1opR)
If larger (g≧r), set 1 to table value pk tk
Add.

(3) If topL and topR both exist at the point of interest hi, and the ratio Ω is 5, or the ratio r is 0,5, then the table value pk ti corresponding to the point of interest h1 is 1
decrease.

(4) In cases other than (3) above, the table value p
Rewrite k ti to -4.

(5) When the temporary peak topR exists only on the right side of the valley, if the ratio r<0.5, subtract 1 from the table value pk ti.

(6) When the temporary peak topL exists only on the left side of the valley, if the ratio Ω is 0.5, subtract 1 from the table value pktj corresponding to the adjacent point hj.

Furthermore, in the peak/trough table smoothing process in step 807, that is, in the peak/trough table pkt, if the distances between the determined valleys are sufficiently close, the smoothing process is performed as follows.

In iwl~N-1, the table value pktl corresponding to the point of interest hi is -2, the table value pk tj of the valley located on the right side of this valley is also -2, and the distance between these valleys (j -i) is smaller than a predetermined threshold, (+) If the frequency of the point of interest hi is greater than the frequency of the adjacent point hj (hi > hj), the table value pk
Set ti to 0 (pkti-0).

(2) If the frequency of the point of interest hi is smaller than the frequency of the adjacent point hj (hi ≦hj), the table value pk tj of the valley located on the right side is set to O (pktj = 0).

Next, a specific example using the above method will be described below.

For example, assume that the feature histogram shown in FIG. 10(a) is obtained. In the figure, the horizontal axis is the color feature amount of brightness or saturation, and the vertical axis is the frequency of the feature amount in the image. When the processing shown in FIG. 8 is performed on this histogram, the table value of the peak/trough table pkt shown in FIG. The transition is as follows along the numbers 8 to 8. As a result of this transition, the numerical values shown in the lower row are determined as the final table values. It should be noted that the positions where each table value in FIG. 4B corresponds to each feature 1A-L of the histogram in FIG. That is, each described table value corresponds to the feature amount above the described position, as shown by the dotted line.

First, left end processing and right end processing of the feature quantity histogram are executed (steps 802, 803), and then temporary peaks and temporary valleys are set (step 804). This setting is performed according to the process of step 804 described above, and each table value obtained as a result is shown in the table numbered 1.

Next, evaluation of valleys based on frequency (step 805) and evaluation of valleys based on adjacent peaks (step 806) are performed according to the process described above.

When the feature amount is a valley of A, only topR exists on the right side of the valley, and the ratio r between the frequency of the feature ff1A and the frequency of the temporary peak on the right side of the valley is 0.5 or less (hi /
lopR<0.5). Therefore, 1 is subtracted from the table value corresponding to feature 11A, resulting in a table value of -
2, the table changes to the table indicated by number 2.

In addition, in the case of a valley of the feature value D, t on both sides of this valley
opL and topR are both present. Furthermore, the ratio p between the frequency of this valley and the frequency of the tentative peak on the left side is less than 0.5 (hi/1opL<0.5). Therefore, 1 is subtracted from the table value pk ti corresponding to the feature quantity, resulting in a table value of -2. Furthermore, the ratio Ω and the ratio r between the frequency of each tentative peak value on both sides of the valley with the feature amount D and the frequency of the valley are larger (Nor). Therefore, by adding 1 to the table value pk tj of the temporary peak (feature ff1E) located on the right side of the valley, the table value becomes 2. As a result, the peak/trough table becomes the table shown by number 3.

In addition, both topL and t are on both sides of the valley where the feature value is F.
opR exists and either the ratio Ω or the ratio r is less than 0.5 (Ω, r<0.5). Therefore, 1 is subtracted from the table value pk tl corresponding to the feature ff1F, making the table value -2. Further, the ratio g is greater than the ratio (Ω≧“).

Therefore, the temporary peak located on the left side of the valley (feature ff1E
), add 1 to the table value pktk, and set the table value to 3.
Make it.

As a result, the table transitions to the table indicated by number 4.

In addition, both topL and t are on both sides of the valley where the feature value is H.
opR exists, and either the ratio Ω or the ratio r is greater than 0.5 (N, r≧0.5). Therefore, the table value pkti corresponding to the feature amount H is rewritten to -4. Further, the ratio ρ is smaller than the ratio r (g<f). Therefore, 1 is added to the table value pk tj corresponding to the temporary peak (feature quantity I) on the right side of the valley.

As a result, the table value corresponding to the feature ml is 2,
The table transitions to the table indicated by number 5.

Furthermore, topL and topR exist on both sides of the valley of feature ff1J, and the ratio ``is smaller than 0.5 (r<0
．． 5). Therefore, the table value pk t corresponding to this valley
Subtract 1 from l. Also, the ratio g is larger than the ratio r (Ω≧
r). Therefore, 1 is added to the table value pktk corresponding to the temporary peak (feature metric) located on the left side of the valley, making the table value 3. As a result, the table changes to one table indicated by number 6.

Further, in the valley of the feature mL, only topL exists on the left side of the valley, and lt N is smaller than 0.5 (1) <0.5. Therefore, 1 is subtracted from the table value pk tj corresponding to this valley. As a result, the table value becomes -2 and the table transitions to the table indicated by number 7.

Next, data is smoothed as described above for the peak/trough table pkt number 7 obtained in this way (step 807). In other words, the distance between the valley h+ of the feature iD and the valley hj of the feature iF located on the right side thereof is smaller than a predetermined threshold, and the table value of the 6 valleys is -2. Furthermore, the frequency of valleys in the feature amount F is greater than the frequency of valleys in the feature amount (hf≦hj). Therefore, the table value pktj corresponding to the valley of the feature amount F is set to 0. As a result, the table finally becomes the table shown at the bottom of FIG. 10(b).

Of this final table, features ff1A,D.

Each point on the histogram corresponding to H, J, and L is found as a valley, but the valleys AL at both ends of the histogram are not considered, and the valley with a table value of -4 (feature M H) is also considered. Must not be. In other words, the effective valley for dividing the data image into the target area and the background area is the table value - 2.
are the valleys corresponding to the double-circled features ff1D and J. Among these valleys, the frequency of valleys close to the area segmentation threshold obtained by applying Otsu's discriminant analysis method to the histogram in FIG. 10(a) becomes the true area segmentation threshold.

By correcting the ambiguous results of Otsu's discriminant analysis method in this way, it becomes possible to perform highly accurate travel route discrimination with few errors.

Next, a description will be given of a method for extracting a driving course by iterative threshold processing using a template image. In addition,
In the following explanation, it is assumed that the autonomous vehicle travels on a course divided by grass, dirt, etc., and a case will be described in which a traveling road area is extracted.

FIG. 11 is a flowchart showing an overview of the algorithm for recognizing the driving route.

First, using a color CCD camera installed in the vehicle,
The scene of the driving route is imaged (step 1101). Then, the captured RGB color image signals are imported into the color camera input device, and the imported RGB original image data is converted into ISH data as described above (step 1102
). Based on the feature image converted to ISH data, a color image processing device creates a histogram representing the characteristics of each feature and its number of pixels as described above (step 11).
03).

Next, based on the created histogram, threshold processing is repeatedly applied to extract the driving route as a binarized image.

Since the image extracted in this way has noise and finely divided areas, the following processing is performed. That is, each area is labeled by a labeling device which will be described in detail later (step 1104). Then, the area and center of gravity of each labeled area are measured, and areas where the center of gravity is above the horizon position calculated from the camera attachment position and areas with small areas are removed (step 1105).

Next, based on the driving path image that was first extracted by the color image processing device, areas that are darker and brighter than the driving path area are found, these areas are merged, and the relationship between each area is determined. Describe (step 1106)
. The drivable range, that is, the road edge, is detected based on this area merging and the description of the area relationship, and the image capture time is added to this drivable range information. Then, the drivable range information is transmitted to the data management unit that constitutes the image processing device (step 1107). After this, the process returns to step 1102 and repeats the above process.

FIG. 12 is a block diagram showing the details of the process of extracting the travel road area.

First, brightness data I and saturation data S are obtained from the RGB data of the captured original image, and each data is stored in the image memory as a feature image (block 1201). Next, a histogram with brightness as a feature quantity is created based on the brightness data stored in the image memory (block 12).
02). The number of divisions of the feature amount, which is the horizontal axis of this histogram, is 40 points, and this number of points is large enough to extract the driving route from a global perspective. Next, the created histogram is normalized (block 1203).

Then, the well-known Otsu's discriminant analysis method is applied to the normalized histogram to calculate a threshold value for distinguishing the driving road region from the background region (block 1204).

On the other hand, based on the shape of the created histogram, peaks with a large number of feature pixels and valleys with a small number of feature pixels are determined as described above, and a peak/trough table, which is a list of peaks and valleys, is created. (Block 1205). The value of each feature amount at each peak and valley is used as a candidate value for threshold processing.

Since the threshold value determined by Otsu's discriminant analysis method in block 1204 may contain an error, block 1204
The threshold value based on Otsu's discriminant analysis method is corrected using the candidate value for threshold processing obtained in step 05. In other words, the threshold value obtained in block 1204 and the candidate value obtained in block 1205 are compared, and the candidate value in block 1205 that is closest to the threshold value in block 1204 is used for separating the driving road region and the background region. Set as threshold (block 12
06). Next, the brightness image is binarized using this threshold value (block 1207).

The brightness image is composed of 512×512 pixels and is expressed as I(i,j).

Here, i+J is an integer that satisfies the conditional expression O≦i+J≦511. In addition, the threshold value found in block 1206 is set as XI, the minimum feature value in the feature range to be divided into regions is a threshold value XO1, and the maximum feature value is a threshold value X2. Note that the initial values of the thresholds XO and X2 are 0 and FF (h e x). Here, if the brightness image 1 (i, j) satisfies the following conditional expression, F F (h e x) is stored in the memory M1ct, j).
(block 1208).

XO≦I (i, j)<XI Also, if brightness image 1 (i, j) satisfies the following conditional expression, write FF (hex) to memory M2 (i, j) (block 1209) .

X1≦l (i, j)<X2 Next, the template image stored in the ROM is read (block 1210). A template area is set within this template image so that the most stable driving route information can be obtained. This area setting is performed based on the angle of depression, 1 angle of view, and focal length of the color camera attached to the autonomous vehicle, and the memory corresponding to the template area is
F(h e x) is stored. Next, the degree of overlap between the read template image and the memories M1 and M2 is calculated (block 1211). That is, the logical product of the template image and the memory M1 and the memory M2 is calculated, and the number of pixels whose logical product is "1" is accumulated for each memory M1 and memory M2.

Next, the ratio of the total number of pixels for each memory to the number of pixels in the template area is determined. It is assumed that the memory information for which this ratio is 50% or more includes the running road area, and further, the memory M1 for which the ratio is 50% or more
Alternatively, the image stored in M2 is repeatedly divided into regions as follows. Further, by selecting a memory based on this ratio, the image is divided into regions by the threshold value X1, and division is also performed on the histogram.

In other words, the memory M1 whose feature value is within the range of XO to X1
This means that the area stored in the memory M2 is divided into the area stored in the memory M2 and the area stored in the memory M2 whose feature amount is within the range of X1 to X2. Further, when the degree of overlap between each memory M1 and M2 and the template image does not exceed 50%, memory M3 in which the sum of memory M1 and memory M2 is stored is selected (block 1212), and based on memory M3, Area segmentation is then performed.

For example, if the degree of overlap between the memory M1 and the template image is high, the area configured by the feature values within X1 from the threshold value XO is set as the road candidate area (block 1
21 B). Then, this road candidate region is further repeatedly divided into regions. The threshold value X1' for this edge-returning region division is the value within the range of the threshold values XO to X1 among the valley candidate values found in block 1205 (
block 1214). Furthermore, if there is no valley candidate value within this range, the repeated division process is not performed.

Minimum value XO of the feature range subject to repeated region segmentation
' is the same threshold value XO as the previous area division,
The maximum value X2' becomes the threshold value X1. Regions constituted by feature amounts within this range are binarized again in block 1207, and then the same processing as the previous region division is performed to perform repeated region division. This repetition is performed until there are no valley candidate values found in block 1205 within the feature amount of the road candidate region to be divided. By repeating the process in this way, the first road image area is finally obtained.

If there is a shadow area that is darker than the brightness of the obtained road image area and a bright high-intensity area, thresholds for determining these areas are set as described below (block 1215). Then, based on this threshold, each region is divided into regions in the same manner as described above to obtain shadow regions and high brightness regions. In addition, if the saturation of the shadow area and the saturation of the road image area are similar based on the processing results based on the saturation images that are performed at the same time, then the logical product of each is taken and the final image is combined into one area. Low brightness area. Similarly, if the calculated saturation of the high brightness area and the saturation of the road image area are similar, the logical product of each is taken and the final high brightness area is determined as one area. Furthermore, the connection relationship between the road image area and the low-luminance area is checked, and if there is a relationship that allows them to be merged, merging processing is performed. Similarly, the connection relationship between the road image area and the high brightness area is investigated.

If there is a relationship that allows merging in this way, merging processing is performed. As a result, if there is a shadow on the driving path, or if your position is in the shadow and there is a high brightness part due to direct sunlight irradiating the driving path far away, by performing this merging process, it will be possible to match the actual driving path. This results in a travel path area having a suitable shape.

Although the above processing was performed on brightness images, similar processing is performed on chroma images as well. However, block 121
Since the processing in step 5 is unique to brightness images,
This process is not executed. In the processing result using this chroma image, if each region has similar colors in which R and GB have similar values, the histogram distribution of each region has a similar shape.

This is the above-mentioned conversion formula from RGB data to saturation S data (% formula %) Therefore, when the six values of R, G, and B are similar, in cloudy skies or on ordinary paved roads, the saturation is It is difficult to distinguish between each region. However, there are color differences, and R, G,
If the B values are not similar, it is possible to distinguish each region even if there is no difference in brightness between the regions. Therefore, in order to accurately extract the road image area under any scene, two feature quantities, brightness and saturation, are used. Then, by taking the logical product of the two types of road images extracted from the brightness image and the chroma image, highly accurate driving road area information can be obtained.

FIG. 13 shows the process of extracting a road candidate region in the road image region extraction process.

First, a feature histogram 1302 is created from a lightness feature image 1301. The horizontal axis of this histogram 1302 indicates brightness, and this brightness ranges from 0 to FF (h e x
) is expressed as a numerical value.

Further, the vertical axis indicates the number of pixels in the original image at each brightness.

Otsu's discriminant analysis method is applied to this histogram 1302 to find a threshold value C for distinguishing between the road region and the background region. Also, based on the shape of the histogram 1302,
Find the peaks and valleys of the histogram and create a peak/trough table. Then, the obtained valley is used as a threshold candidate value for the iterative threshold processing. This histogram 1
In 302, feature quantities a, b, d, e, and f are candidate values.

The threshold value C obtained by Otsu's discriminant analysis method is corrected by the candidate value obtained from the peak/trough table. In other words, the candidate value closest to the threshold value C is set as the corrected threshold value. Then, by using this threshold value, the feature image 1
301 is binarized.

As a result, the divided image 13 of feature amount 00 (hex) ~ b
03 and divided image 130 with feature amount b-FF (hex)
4 is obtained. At this point, each divided image 130
3. It is not known which image of 1304 includes the driving road area.

For this reason, the template image 130 assuming the road position
5 and each image 1303 and 1304 to calculate the degree of overlap with each image. A trapezoidal template area is set at the bottom of the template image 1305 as shown. As described above, FF (hex) is stored in the memory element corresponding to this template area, and 00 (hex) is stored in the memory element corresponding to the background area of the template area. In the case of this example, the image 1304 has the largest number of overlapping parts with the template image 1305. Therefore, based on the calculation result of the degree of overlap, it is determined that the image 1304 with the feature amount b to F F (h e x) includes the driving road area, and the road candidate area image 1306 corresponding to the image 1304 is can get.

Since there are still other candidate values remaining between the feature values of b-FF (hex), next, this road candidate area image 1
Area division is performed for 306.

That is, the road candidate area image 1306 is binarized using the threshold value d. Through this binarization, a divided image 1307 with feature amounts b to d and a divided image 1308 with feature amount d-FF (hex) are obtained. Next, each image 130 obtained
7. The degree of overlap with template image 1305 is calculated for 1308 in the same manner as described above. In this example,
Image 1308 has a higher degree of overlap with template image 1305, so image 1 with the feature amount d-FF (hex)
It is determined that image 308 includes a driving road area, and a road candidate area image 1309 corresponding to image 1308 is obtained.

There are also other candidate values e between the features of d-FF (hex).
remains, the road candidate area image 1309 is binarized using this threshold value e. With this binarization, special y! ! ,
A divided image 1310 where i is d - e and the feature amount is e - FF
(hex) divided images 1311 are obtained. and,
The degree of overlap between each image 1310, 1311 and the template image 1305 is calculated in the same manner as described above. In the case of this example, since the image 1310 has a higher degree of overlap with the template image 1305, it is determined that the image 1310 with the feature amount de contains the driving road area, and the image 1310
A road candidate area 1312 corresponding to is obtained.

Since there are no other candidate values remaining between the feature values d - e, this road candidate region 1312 becomes the final binary image of the road region. Although the above processing was performed on the brightness image, similar processing is performed on the chroma image as well. after that,
The final driving path area is obtained by performing a logical product of the driving path area extracted from the brightness image and the driving path area extracted from the saturation image.

A common method is to binarize an image using a threshold and divide the image into regions. However, as in this example, by calculating the degree of overlap with the segmented images using a template image that takes into account the position of the driving route, the threshold value for the next region segmentation performed on the histogram is determined. This process can be performed quickly and simply. As a result, a travel road area matching an actual travel route can be obtained quickly, easily, and at low cost.

Furthermore, the brightness of the scene captured by the camera may change as follows. For example, the brightness may change because the direction of movement of your army changes due to a curve, or the brightness may change because the weather becomes sunny or cloudy. In such a case, the feature value histogram does not always show a constant shape, so region segmentation using a fixed threshold value does not provide a correct driving path region. However, with this method, even if the state of the input image, that is, the brightness, changes from moment to moment, it is possible to always accurately extract the travel road area. This is because the present method finds the feature quantity distribution with the highest degree of overlap with the template image, and sets the optimal threshold value each time according to the feature quantity at that time for images that change moment by moment.

Conventionally, in image processing that performs region division, an original image is divided into a plurality of regions, and identification processing is performed on the divided images to verify the validity of roads.

However, in this method described above, area segmentation is performed using threshold processing in order to extract the target object (road). In this way, in the conventional method, the object was verified from the processing results, but in this method, the processing approach is reversed in that the object is the object from the beginning of the processing. Therefore, it is possible to efficiently extract the road area.

Next, we will explain ``a method for extracting shadows and high-brightness areas from a driving course that focuses on differences in brightness''. This method has already been briefly explained in the above-mentioned "Travel course extraction method by iterative threshold processing using a template image," and this method will be described in detail below.

For the brightness image, by applying the above-mentioned "driving course extraction method using iterative threshold processing using a template image" and "threshold setting method based on the shape of the feature histogram in iterative threshold processing", A histogram showing the distribution of features of the driving course was obtained. This method determines areas darker than the road area and areas lighter than the road area based on this histogram. Further, this method is processed within a color image processing device.

FIG. 14 shows an overview of processing when this method is applied to each case in which various input images are captured. Case 1 is a case where only a driving course with negative road surface conditions is captured as an input image.

In this case, the own vehicle is located in front of a curve on a road that curves to the right. The road candidate area extracted in this case 1 has the same shape as the input image. This is because the input image is only a --like driving course, and therefore low-luminance areas and high-luminance areas are not extracted by this method.

Case 2 is a case where an input image is captured in which there is a partial shadow on the road surface of the driving course and a high brightness part due to reflected light or the like is formed in the far part of the driving course. In this example, the road curves to the right, and before the right curve there is a branch road that curves to the left.

The vehicle is located in front of these curves. The road candidate area extracted in this case 2 has a shape in which dark areas with shadows and bright areas with high brightness are excluded. In addition, shadow parts that appear on the near side of the road can be individually extracted by extracting low-luminance areas using this method. Further, high brightness areas due to reflected light that are formed far away from the road can be individually extracted by extracting the high brightness areas.

Case 3 is a case where an input image in which shadows from trees are formed on the road surface of the driving course is captured during weather with strong sunlight, and the road surface is illuminated by sunlight filtering through the trees. In this example, the own army is located in front of a curve on a road that curves to the left. The road candidate area extracted in this case 3 has the same shape as a shadow created by sunlight filtering through the trees. This is because the shadow portion has a high degree of overlap with the template image due to strong sunlight. Therefore, low brightness areas are not extracted by this method. Further, distant road areas and areas where sunlight shines through the trees are extracted as high brightness areas due to strong sunlight.

Case 4 is a case where a band-shaped discolored portion is formed along the side strip of the road, for example, when the road is discolored due to construction of a paved road or the like. In this example, the own car is
It is located in a position where you can see the discolored area on the straight road on your right. The road candidate region extracted in this case 4 has a shape in which this discolored portion is excluded. Furthermore, since this discolored portion has higher brightness than the road area, it is extracted as a high-brightness area by this method. Furthermore, since there are no shadows on the road surface, low-luminance areas are not extracted.

Next, the details of this method will be explained below using Case 2 mentioned above as an example.

FIG. 15 is a histogram created based on the input image captured in case 2.

The feature amount of this histogram is brightness, and this brightness is shown on the horizontal axis. Moreover, the vertical axis is the number of pixels at each brightness.

The feature value range in part A of the figure is the range that includes the driving road area, and the range that includes the pixels most corresponding to the driving course in the aforementioned "driving course extraction method by iterative threshold processing using a template image". This is the part that is said to be. Further, range A is a range in which the feature amount is from tl to t2, and is divided into regions using each feature ff1tl and t2 as thresholds. Furthermore, the portion B on the left side of the feature amount t1 forms one mountain sandwiched between valleys, and has a feature amount distribution in a dark range with lower brightness than the portion A. The feature amount range of this part B is from t3 to tl, and each feature mt3 and t] becomes a threshold for region division. Further, the C portion on the right side of the feature amount t2 has a feature amount distribution in a bright range with higher brightness than the A portion, and forms one mountain sandwiched between valleys like the B portion. The feature amount range of this C part is from t2 to t4, and each feature ff1t2
and t4 becomes a threshold for region division.

Note that portions B and C shown in the figure form one mountain, but a feature frequency distribution that does not form a single mountain like this forms a meaningful region for region segmentation on the image. It cannot be recognized as a distribution. Therefore, such a feature quantity distribution is not selected as a region extraction target as it is not a dominant distribution. In the illustrated example, both the B portion and the C portion have a dominant distribution, but the present method is applicable even if only one is a dominant distribution.

The feature quantity distribution of part A corresponds to the road candidate area of case 2 shown in FIG. Further, the feature amount distribution of portion B corresponds to a shadow region having a lower brightness than this road candidate region, and the feature amount distribution of portion C corresponds to a high brightness portion having higher brightness than the road candidate region. This method extracts each area corresponding to each feature value distribution of the B part and C part adjacent to the A part.

First, in order to extract a region corresponding to part B of the histogram, an input image whose feature quantity is brightness is binarized using thresholds t3 and tl. Furthermore, in the aforementioned "driving course extraction method using iterative threshold processing using a template image", a road area image is obtained based on an input image that uses saturation as a feature. This road area image is obtained by first dividing the original image into areas, and is an image in which dark areas such as shadows and high brightness areas are included in the road area. 1 is stored in the memory element corresponding to this road area, and 0 is stored in the memory element corresponding to the other background areas. Further, in the above binarized image, 1 is stored in the memory element corresponding to the pixel area whose brightness is from t3 to tl,
0 is stored in memory elements corresponding to other areas.

Therefore, by performing a logical product of the road area image and the binarized image, only dark areas such as shadow parts on the driving road area are individually extracted.

In addition, in order to extract the area corresponding to part C, the threshold value t2 and ti
The brightness image is binarized by

Then, the road area image extracted from the saturation image and these two
High brightness areas are individually extracted by performing a logical product with the valued image in the same manner as in the case of extracting the B portion described above.

Next, a method for determining the threshold values t3 and ti required when dividing portions B and C into regions will be described. Although the peak/trough table shown in FIG. 10(b) was obtained using the above-mentioned "threshold setting means based on the shape of the feature histogram in repeated threshold processing," the histogram shown in FIG. In the same way, a peak/trough table (not shown) is obtained. Each table value in this peak/trough table is expressed as pktI as in FIG. 10(b).

It is assumed that the subscript i changes from i-0, 1, 2, . . . N, N-1 in order from the origin of the graph corresponding to each table value.

In the histogram of FIG. 15, the table value corresponding to the threshold value t1 is assumed to be pk tj. And peak
In the valley table, look at each table value from this table value pk ti to the left, and find pk'tk (pktk<O) where the table value becomes negative by pk tO.
Determine whether or not there is. A feature point with a negative table value is a point corresponding to the bottom of a valley. There is pk tk that becomes negative by pk tO, and furthermore, from the feature point of pktj to p
When the distance to the feature point k tk is smaller than the threshold value (pk tj −pk tk <L threshold), the feature amount corresponding to this table value pktk is set as the threshold value t3. Furthermore, if there is no pk tk that becomes negative by pk tO, or if the distance exceeds the threshold, it is assumed that there is no dominant region darker than the road region.

The feature quantity corresponding to the threshold value t4 on the histogram can be obtained in the same manner as the method for obtaining the feature quantity corresponding to the threshold value t3. In other words, if the peak-trough table value corresponding to threshold t2 is pktra, then this pk
Looking at each table value from tm to the right, p k t
pk tn (pk tn <
O) is present or not. If pk tn becomes negative by p k t N-1, and the distance from the feature point of pk tn to the feature point of pktm is smaller than the threshold (pktn - pktIll<L threshold), this table The feature amount corresponding to the value pk tn is set to the threshold value t4
shall be.

Furthermore, if there is no pk tn that becomes negative by p k t N-1, or if the distance exceeds the threshold,
It is assumed that there is no dominant region that is brighter than the road region.

As described above, this method is a "threshold setting means based on the shape of the feature histogram in iterative threshold processing"
By using the peak/trough table obtained in , it is possible to individually extract shadows, high-intensity areas, and discolored areas even if the road surface conditions change due to changes in the weather or pavement construction. .

Next, the "labeling device" will be explained in detail below. [
The image extracted by the method for extracting a driving course by repeated threshold processing using a template image has noise and finely divided regions. For this purpose, each region of the extracted image is labeled by a labeling device, and the validity of each labeled region is determined. Based on the results of this labeling process, areas where the center of gravity is above the horizon position and unnecessary small areas caused by noise or the like are removed.

FIG. 16 is a block diagram showing a schematic configuration of this labeling apparatus, and FIG. 17 is a general flowchart showing an outline of this labeling process.

First, an image extracted by the color processing device is loaded into the input memory 1601 via the image bus (NE Bus) (step 1701). This image information is 512x
This is 512x8 bit information, and this is used as a multivalued original image input. Next, primary labeling is performed using the run-based temporary labeling method described later (step 1702).
. This primary labeling is performed by a labeling processor (KL).
P) 1602. Line buffer memory (LBM) 1
603 and label matching memory (LMM) 160
It is mainly executed in the 4th class.

After this primary labeling, the label matching memory LM
Preprocessing is performed to organize the data arrangement of M1604 (step 1703). After this preprocessing, secondary labeling is performed and at the same time feature quantities such as the area and center of gravity of each region are extracted (step 1704). The processing in steps 1703 and 1704 is mainly executed in the feature extraction processor KLC1605. Through this secondary labeling, a label attached to a pixel located at each address is stored in a label memory (LABELM) 1606. At the same time, the area and center of gravity of each extracted region are stored in the feature memory 1607. After this, LABELM16
The label image information stored in 06 is output to the NE BUS (step 1705).

The number of KLP1602s used for label generation is ], and the number of LBM160s used is the line register for run processing.
3 is configured using four label memories KLM, which will be described later. Additionally, the maximum number of temporary labels for LMM1604 is 102.
3, it is configured using eight KLMs. When the maximum number of temporary labels is 4095, 32 KLMs are used and LM
Configure M1604.

Conventional labeling was performed only when the input image was a binary image, but labeling using this method
By using 602, it is possible to label multivalued images as well. In other words, it is possible to label several types of images at once. For example, as shown in FIG. 18, assume that three types of binary input images 1801, 1802, and 1803 are input. These binary images are added together to form a 512x512
It is converted into a multivalued image 1804 of x8 bits. This conversion process is performed as pre-processing of the labeling process. A multivalued input image 1804 is labeled under the control of a labeling processor 1807 (KLP1602) using a line register 1805 (LBM1603) and a label matching memory 1806 (LMM1604).

In this labeling, organize the labeling of each area,
Final label image 18 of 512 x 512 x 12 bits
Output as 08. This labeling processor 180
7 (KLP1602) enables labeling processing for multivalued images, reduction of the number of temporary labels using runs, and single-scan labeling.

Each pixel of the multivalued input image is labeled according to each pixel value under the control of the KLP 1602. and,
Region segmentation is performed based on the connection relationship between pixels having the same label value, and a new label is generated based on this connection relationship.

For example, conventionally, when an input image has a region made up of step-like pixels shown in FIG. Label generation was performed by scanning the . As a result, when attaching temporary labels, three types of temporary labels, 1 to 3, are required as shown in the figure.

However, according to the labeling method using runs according to the present method, even if there is an input image shown in FIG. 19(b) consisting of stepped pixels similar to FIG. The number of temporary labels can be reduced. In other words,
As shown in FIG. 3C, each pixel is divided into rows called runs along the rask scan. In the illustrated case, it is divided into two run 1 and run 2. The labeling processor KLP writes a flag in the line buffer memory LBM until the scanning of the run reaches the last pixel, judges the temporary label, and at the last pixel of the row, connects all pixels of the run to the connection relation between each pixel. Write a temporary label that takes into account.

As a result of scanning and performing the above processing on run 1 shown in the same figure (C), the pixels corresponding to run 1 are shown in the same figure (d).
The temporary labeling shown in is performed. This temporary label "1"
' labeling is done simultaneously for each pixel. this is,
This is because the labeling memory KLM, which will be described in detail later, is used as the memory. By subsequently scanning run 2, the provisional labeling shown in FIG. 4(e) is performed. Since run 2 is connected to the temporary label "1" of run 1, when the last pixel of run 2 is scanned, the temporary label "1" is simultaneously written to each pixel of run 2.

By labeling using runs in this way, the same figure (b)
The stepped pixel labeling shown in can be performed using one type of label "1", and the number of temporary labels is reduced. In other words, the scale of the labeling circuit can be reduced by grouping pixels into one group called a run and performing processing on a run-by-run basis.

Next, details of the labeling process by the labeling processor KLP will be explained. Labeling is performed by scanning each pixel along each run with the window shown in FIG. By scanning this window along each run, the pixel of interest appears in the T (target) section, the pixel located above the T section appears in the a section, and the pixel on the right side of the T section appears in the b section. Pixels located next to each other appear. Below, T.

Let a and b indicate label values of the input image appearing in each part. Note that when the window is scanning in the middle of a run, the value of the flag is output to the line buffer memory LBM as an output label, and when the window reaches the last pixel of the run, all flagged pixels are Label is written.

The internal structure of KLP is shown in the block diagram of FIG. KLP selector 2101. Temporary label register T
m12102. It is composed of a counter Cnt 2103, a first comparison circuit 2104, and a second comparison circuit 2105. The first comparison circuit 2104 has a label value T,
a and b are given, and a multi-value comparison of input images is performed. The result of this comparison is the select terminal 5ell of the selector 2101.
given to. The second comparison circuit 2105 includes a value Mat(a) of the label matching memory LMM of the label value a,
Value of temporary label register Tm12102 and counter C
The value of nt2103 is given, and at the same time the 6 value is selector 2
101 terminals A and B.

given to C. The value of Tm12102 is determined by the output signal from selector 2101. Also, a line buffer memory LB is connected to the terminal of the selector 2101.
The value of the flag FLAG stored in M is given.

The second comparison circuit 2105 compares these six applied values. The connection relationship between each label is determined based on the comparison result, and the comparison result is transmitted to the select terminal 5 of the selector 2101.
Output to elO.

The selector 2101 outputs a matching address MAT ADDR and matching data MAT DATA based on the six supplied values, and changes the storage contents of the label matching memory LMM.

At the same time, the selector 2101 outputs a temporary label value and an output label value (LABEL).

22 to 27 show flowcharts of window processing.

FIG. 22 shows the 6-value T in the first comparison circuit 2104,
A flowchart is shown when performing a comparison judgment process between a and b. First, it is determined whether the label value T of the pixel of interest is equal to 0 (step 2201). If T is 0, processing 1 described later is executed (step 2202). If T is not 0, label value T and label value S are compared (step 2203). When label value T and label value S are equal, label value T and label value a are compared (step 2204). If the label value T and the label value a are equal, process 2 is executed (step 2205). In other words, process 2 is a process executed when the six values Ta and b are equal. In this case, if each label value is expressed as a circle, the window state will be the state drawn adjacent to the processing box shown in step 2205.

If label value T and label value a are not equal, process 3
(step 2206). In other words, process 3 is a process that is executed when label value T and label value S are equal and label value T and label value a are different. In this case, if label value T and label value S are expressed as O, and label value a is expressed as x, the window state will be the state drawn adjacent to the processing box shown in step 2206. After the processing in step 2205 or step 2206, a flag is set at the position of window T in the line buffer memory LBM 1603 (step 2207).

Further, in step 2203, even if label value T and label value a are not equal, label value T and label value a
(step 2208). If the label value T and the label value a are equal, process 4 is executed (step 2209).

In other words, in process 4, label value T and label value a are equal,
This is a process executed when label value T and label value S are different. In this case, if label value T and label value a are expressed as O, and label value S is expressed as x, the window state becomes the state drawn adjacent to the processing box shown in step 2209.

If label value T and label value a are not equal, process 5
(step 2210). In other words, in process 5, label value T and label value a are different, and label value T
This is the process that is executed when the label value and the label value are different.

In this case, if label value T is expressed as O and label value a and label value S are expressed as ×, the window state is set to step 2.
It is drawn adjacent to the illustrated processing box 210 . Then, step 2209 or step 221
After processing 0, the temporary label register Tml in KLP is cleared (step 2211). This Tml stores a label for a pixel that was in the T portion immediately before reaching the current window position.

The flowcharts in FIGS. 23 to 27 described below show flowcharts of the comparison judgment processing from processing 1 to processing 5.

FIG. 23 shows a flowchart of the above-mentioned process 1. Process 1 ends without executing anything.

FIG. 24 shows a flowchart of the above-mentioned process 2. First, the label value of the previous pixel stored in the temporary label register Tml is compared with 0 (step 2401). Tml
When the label value of is equal to 0, the value Mat(a) of the label matching memory (LMM) 1604 of the label value a
is written into the temporary label register T m l 2 ], 02 (step 2402). If the label value of Tml is not equal to 0, the label value of Tml and the counter Cnt
Compare with the counter value of 2103 (step 2403
). The counter 2103 stores the value of the newest label.

If the label value of the temporary label register Tml and the counter value of the counter Cnt are equal, the value Mat(a) of the label matching memory (LMM) of the label value a is written to Tml (step 2404), and the value of Tml is written. If the value and the value of the counter Cnt are not equal, the value Mat(a) of LMM and the value of Tml are compared (step 24
05), If the value Mat(a) of LMM and the value of Tml are equal, nothing is executed (step 2406), L
If the value of MM Mat(a) and the value of Tml are not equal, the smaller value of the value of Tml and the value of LMM Mat(a) iMin (Tm 1. Mat (
a) ) Write l to Tml. Furthermore, the smaller value fM i n (Tm 1. Ma t
(a)) Label matrix 2407) with a label value equal to the larger of the two values.

FIG. 25 is a diagram showing a flowchart of the above-mentioned process 3. In process 3, first, the value of the temporary label register Tml is compared with 0 (step 2501). Tml value is O
, the counter value of the counter 2103 is set to T
ml (step 2502). Further, if the value of Tml is not equal to 0, nothing is executed (step 2503).

FIG. 26 is a diagram showing a flowchart of the above-mentioned process 4. First, the value of Tml is compared with 0 (step 26
01). If the value of Tml is equal to 0, the label value a
The LMM value Mat(a) is set as the label value of the target area, and this Mat(a) is written into the T section and all registers with flags set (step 2602). At this time, all LBM flags are cleared. Further, if the value of Tml is not equal to 0 in step 2601,
Compare the value of Tml and the value of Cnt (step 260
3). If the value of Tml and the value of Cnt are equal, the LMM value Mat(a) of label value a is set as the label value of the target area, and this Mat(a) is stored in all registers where the T part and flag are set. (step 260)
4).

At this time, all LBM flags are cleared.

Furthermore, if the value of Tml and the value of Cnt are not equal, the value of Tml and the value of Mat(a) are compared (step 2605). Then, when the value of Tml and the value of Mat(a) are equal, the value of LMM of label value a Mat(
With a) as the label value of the target area, this Mat(a) is written to the T section and all registers with flags set (step 2606). At this time, all LBM flags are cleared. Also, the value of Tml and Mat(a)
If the values of Tml and label value a are not equal, the value of Tml and the label value a
The smaller value fM i of the LMM value Mat(a) of
n (Tm I, Mat (a)) Write l as the label value of the target area to the T section and all registers where the flag is set. At this time, all LBM flags are cleared. Furthermore, the smaller value (Min (T
m l, Mat (a) ) 1 is defined as a label matching memory [Mat(Min (Tm l, M
a t (a) ) l Write in (Step 2)
607).

FIG. 27 is a diagram showing a flowchart of the above-mentioned process 5. First, the value of the temporary label register Tml is compared with 0 (step 2701). If the value of Tml is equal to 0, the value of Cnt is used as the label value of the target area, and this Cn is stored in the T part and all registers where the flag is set.
Write the value of t. At this time, all LBM flags are cleared. Further, the value of Cnt is written to Mat(Cnt) with a label value equal to the value of Cnt, and the value of Cnt is counted up by one (step 2702).

Also, if the value of Tml is not equal to 0, the value of Tml is written to the T part and all registers where the flag is set, using the value of Tml as the label value of the target area (
Step 2703). At this time, all flags of the LBM are cleared. Next, the value of Tml and the value of Cnt are compared (step 2704). If the value of Tml and the value of Cnt are equal, the value of Cnt is written to Mat(Cnt) of the label value equal to the value of Cnt. Furthermore, Cn
Count up the value of t by 1 (step 2705)
, and if the value of Tml and the value of Cnt are not equal, nothing is executed (step 2706).

Next, the labeling memory KLM used in the line buffer memory LBM and label matching memory LMM will be explained. In conventional memories, data could only be written to one internal register with one address specification. However, by using the KLM described below, data can be written to multiple internal registers at once. For this reason, this labeling memory KLM consists of a line register (LBM) for run processing, a label matching memory (LMM) that does not require label integration, a label image memory (LABELM) for one scan, and a label matching memory that can perform label matching. It can be used.

KLM is composed of multiple registers, but the second
FIG. 8 shows the block configuration of one of these registers. This block is one unit of the KLM structure. Comparators 2802 are connected to each register 2801 in pairs. This comparator 28021: is given output data DATA from the register 2801 and information COM to be compared with this output data. Comparator 2802 compares the given data and sends the comparison result to AND circuit 2803.
Output to. In addition to this, the output of an address decoder circuit 2804 is given to the AND circuit 2803. AND circuit 2803 outputs a signal if either comparator 2802 or decoder circuit 2804 outputs a signal.
A signal is output to OR circuit 2805.

A write signal WR from the CPU is applied to the OR circuit 2805, and an enable signal is applied to the register 2801 in synchronization with the write signal WR. In other words, whether it is selected by the address decoder circuit 2804 or
If the comparison result in comparator circuit 2802 matches, data is written to register 2801 in synchronization with write signal WR. Addressing each decoder circuit 2804 and comparison judgment in each comparator circuit 2802 are all performed simultaneously. Therefore, it is possible to simultaneously rewrite the contents of a plurality of registers constituting the KLM by one addressing or one data comparison decision.

The temporary labels obtained by the labeling process using the runs described above become discontinuous values when the labels are integrated. The contents of the label matching memory LMM at this time are shown in FIG. 29(a). Labels 1, 3.6 are stored as data corresponding to pixels at addresses 1 to 10. Since this label value is a discontinuous value, the same figure (b)
A feature extraction processor KLC shown in FIG. 1 and described in detail later converts the label into a label having continuous values as shown in FIG. That is, the label values stored in the label matching memory LMM configured by KLM are 1, 2.
．． It becomes a continuous value of 3.

More specifically, the labeling processor KLC generates an address for the LMM and takes in the data indicated by the corresponding address. Subsequently, the fetched data and address are compared, and if they are the same, new data is output to the LMM and the label value is rewritten.

If each location is different, the next address is output to the LMM and the data is compared with the next address. Thereafter, by repeatedly executing this process, the image shown in FIG. 29 (a
) are converted into continuous label values shown in (C) of the same figure.

Specifically, since the data (1) at address 1 in FIG.
Rewrite the data at addresses 2, 4, 5°7 with new data 1. In the case shown in the figure, the old data and new data happen to be the same 1, so the figure (a
) and (c) of the same figure, there is no change in the data at the corresponding address.

Next, data (1) at address 2 is compared with the address. Since the address and data are different, the next address 3
occurs. Then, data (3) at address 3 is compared with the address. Since the address and data are the same, 2 is output as new data. LMM is KLM
Therefore, the data at addresses 9 and 10 that have the same data (3) as the data at address 3 are shown in the same figure (
As shown in C), all are rewritten to 2 at the same time.

The KLC then generates a new address 4. Since the address and data are different, address 5 is generated next, and thereafter, the same process as above is repeated. As a result, discrete values are converted to continuous values.

FIG. 30 is a block diagram showing the internal configuration of this feature extraction processor KLC. By using this KLC, preprocessing of temporary labels generated by primary labeling is performed. In addition, during secondary labeling, at the same time as labeling, calculation of the area of the same label area, calculation of the sum of the X-direction addresses of the same label area, and calculation of the sum of the Y-direction addresses of the same label area are executed by this KLC. .

KLC has a +1 adder 3001 and two adders 300
2.3003, a comparator 3004, and two counters 3005 and 3006. The +1 adder 3001 increments the count by one when there is a pixel input having the same label value, calculates the area of the same label area, and outputs it as 5ize. Adders 3002 and 3003 contain X-direction address values.

A Y-direction address value has been input. Then, when there is a pixel input with the same label value, the address values are added in each direction, and the sum of the address values of the same label area is calculated for each address direction. Each total value is X A
Output as DDR and Y ADDR. By dividing the total value of addresses in each direction by the area of the same label area, the center of gravity in each direction can be determined. and,
The center of gravity of each identically labeled area is determined, and areas whose center of gravity is above the horizon are removed as not being effective for extracting road candidate areas.

The comparator 3004 and counters 3005 and 3006 are
It is used for preprocessing of temporary labels that occur during primary labeling, that is, for converting discontinuous label values into continuous label values. A clock signal CLK is input to counters 3005 and 3006, and the output of counter 3005 becomes a matching address MAT ADDR that is output to label matching memory LMM. This address is also given to comparator 3004 at the same time. Also, counter 30
The output of 06 becomes data MAT DATA which is sent to the label matching memory and output to MM.

The comparator 3004 compares the given address and data as described above, and outputs a signal to the counter 3006 if the six values of the address and data match. When the counter 3006 receives this signal, the Mat
Outputs the current counter value to DATA and increments the value by one.

The processing time of the labeling process described above required the sum of the rask scanning time of two scans performed along each run and the label integration time. However, the 16th
By configuring the label memory LABELM 1606 shown in the figure using the labeling memory KLM described above, labeling processing can be executed in one scan. That is, as shown in FIG. 31, when a multivalued input image 3101 is captured, the labeling processor KLP 3102 uses the line register 3103 and the label matching memory 3104 to execute the labeling process in the same manner as described above. . The label value of each pixel obtained by this labeling is written as is into the label image memory 3105 at the time of scanning the last pixel of each run. Note that since label integration is not performed, the values of each label are stored as discontinuous values.

As mentioned above, in the labeling memory KLM, a comparator is connected to each internal register in pairs, and new data is written to the register whose comparison result by the comparator matches and the register whose address has been selected at - degrees. It is something. With this KLM feature 1
Scan labeling is now possible. Furthermore, the labeling processor KLP is a hardware implementation of the algorithm of this method, and enables high-speed labeling.

This one-scan labeling method makes it possible to reduce the labeling processing time to less than half of the previous one. For example, if the device is operating with an 8 MHz clock signal, the traditional labeling processing time requires 66 m5ec for two window scans, plus the time required for label integration. did. However, according to this one scan method, 33m5ec, which is less than 1/2
The labeling process can be performed with .

Further, although normal pixel scanning is performed from the upper left to the lower right of the screen of the input image, in image processing for extracting a road area, the target image is located at the bottom of the image. Therefore, when scanning a target image, there is a possibility that the memory that stores temporary labels will overflow. Therefore, by scanning from the lower right to the upper left of the screen and scanning the target image first, the target image can always be obtained. In other words, if you acquire the target image first, even if memory overflows, the image scan at the time of the overflow will be a scan of unnecessary image parts, and you can always acquire the target image. .

Next, a "method for merging multiple regions" obtained by dividing an image will be described.

Road candidate areas are determined by the “driving course extraction method using iterative threshold processing using template images”.
Shadows and high-brightness areas, which are low-brightness areas, were found using ``a method for extracting shadows and high-brightness areas on a driving course that focuses on differences in brightness.'' Using this road candidate area as a reference, low-luminance areas and high-luminance areas that are strongly connected to the road candidate area are merged into the road candidate area, treated as one area, and regarded as a driving course. To do this, the common boundary adjacent to each area and the perimeter of each area are determined, and the relationship between the connected areas is described. The merging relationship is expressed by the description of the relationship between each region, and the same result as merging regions can be obtained without changing labels as in the conventional method. Note that the common boundary is determined based on the length of the side of the pixels adjacent to each region, and the perimeter is determined based on the length of the side of the outermost pixel of each region.

The algorithm of this method is shown below. There are two methods for this method. Firstly, there is an inverted character mask scanning method in which a mask of inverted characters is scanned. This method is a simple algorithm and is suitable for hardware implementation.

Secondly, there is a region boundary search method that locally searches the boundaries of a region, and this method is effective because it searches only the boundaries of a necessary region, so it can be processed with a small amount of memory.

The first method, the reverse character mask scanning method, is executed by scanning the reverse character mask shown in FIG. 32 from left to right and from top to bottom of the label image. The X pixel appearing in the illustrated mask is the pixel of interest, the a pixel is a pixel located above the pixel of interest X, and the b pixel is a pixel located to the left of the pixel of interest X. For example, assume a pixel area shown in FIG. 33. The mouth shown in the figure represents one pixel, and the numerical value inside the mouth represents the label value of that pixel. In this example, the area with label value 1 and the area with label value 4 are adjacent to each other. Although not shown in the figure, the label value of the background area is 0.

The perimeter ob of each region is determined by scanning the inverted mask along the rows of each region, that is, along the runs. Specifically, the perimeter obl of the area with the label value 1 is determined as follows. First, the X pixel of the inverted character mask is aligned with the pixel located at the left end of the top run. In this case, since pixel a and pixel b are in the background area and have a label value of 0, it can be seen that the two sides of this pixel are located around the area with the label value of 1. Therefore, 2 is added to the counter that counts the perimeter obl. Next, the reverse character mask is scanned to the right, and the X pixel is aligned with the pixel on the right. In this case, the a pixel is in the background area, and the b pixel is in the main area with a label value of 1, which is the sum of the previous mask. Therefore,
It is found that one side of this pixel is located around the main area with the label value 1, and 1 is added to the counter. By scanning the inverted character mask along the run for the area with the label value 1 in this manner, the perimeter obl becomes 18. Note that the unit of the perimeter ob is the number of sides of a pixel.

The perimeter ob4 of the area with label value 4 is also the perimeter obl above.
It is obtained in the same way as , and the value is 2o.

Further, the common boundary cb between the area with label value 1 and the area with label value 4 can be found as follows. This common boundary cb is the label value 4 seen from the area with label value 1.
The common boundary cb14 with the area of label value 4 may be different from the common boundary cb41 with the area of label value 1 viewed from the area of label value 4. In other words, since it has the shape of a mask or an inverted character, and only two neighboring pixels of each pixel are considered, a difference occurs in the common boundary, but this does not pose an equivalent problem when merging multiple regions. Note that if the mask to be scanned is made into a cross shape that looks at four neighborhoods of the pixel of interest, there will be no difference between the common boundaries.

The common boundary with label value 4 seen from the area of label value 1 c
b14 is determined by mask scanning when determining the perimeter obl. This common boundary cb14 becomes 0. That is, even if the inverted character mask is scanned for each run in the area with the label value 1, the pixels with the label value 4 do not appear in the a and b pixels.

The common boundary cb41 between the area of label value 1 and the area of label value 1 viewed from the area of label value 4 can be found by mask scanning when calculating the perimeter ob4. This common boundary cb41
becomes 3. In other words, if the mask is located at the first pixel of the second and third runs in the area with label value 4, b
A pixel with a label value of 1 appears in the pixel. Therefore, by mask scanning for the runs up to the third stage, the common boundary is c
The value of the counter that counts b41 becomes 2.

Further, when the mask is located at the beginning of the fourth run, a pixel with a label value of 1 appears at pixel a of the mask.

Therefore, 1 is added to the counter, the integrated value of the counter becomes 3, and the common boundary cb41 becomes 3. Note that the unit of the common boundary cb is the number of sides of a pixel.

Next, the values of the common boundary between each area and the perimeter of each area determined in this way are stored in an inter-label boundary length matrix shown in FIG. 34.

The numbers 0, 1, 2°, etc. attached to each column of the same matrix
, j, ... kl and the numbers 0, 1゜2 attached to each line
, . . . , i, . . . Ω indicate label values. A number m stored in the location specified by the number in each column and row
ij is the common boundary between the area of label value j and the area of label value j as seen from the area of label value i. Further, the numerical value 2 stored in the k+]th column is the perimeter of the area of the label value l.

Taking the pixel area shown in FIG. 33 as an example, the label value 1
The common boundary with the area with label value 4 seen from the area is cb1
Since the value of 4 is 0, 0 is stored in the memory located in the 1st row and 4th column of the same matrix. Also, the common boundary with the area with label/value 1 seen from the area with label value 4 is cb41
Since the value of is 3, 3 is stored in the memory located in the 4th row and 1st column of the same matrix.

After creating a matrix by finding the perimeter of each region and the common boundary between each region, the degree of connectivity between each region is calculated.

This degree of connectivity is determined by the following equation based on the common boundary and the perimeter.

Connectivity - X/min (A, B) where A is the perimeter of the region with label value i, B is the perimeter of the region with label value j, and m1n(A.

B) represents the smaller of A and B. Moreover, X is the average value of the common boundaries between each region expressed by the following equation.

X-(X1+X2)/2 Xl is the common boundary with the area of label value j seen from the area of label value i, and X2 is the common boundary with the area of label value i seen from the area of label value j. .

In the above formula showing X, both Xl and X2 are not 0 (X
i≠0. This is valid when X2≠0).

When X2 is O(X2-0), X is represented by the following formula.

-X 1 When Xl is 0 (Xi-0), X is represented by the following formula.

-X2 If the calculated degree of connectivity is greater than or equal to a predetermined threshold, the connection between the region with label value i and the region with label value j is strong, and the regions should be merged. Furthermore, if the degree of connectivity is below a predetermined threshold, the connection between the region with label value i and the region with label value j is weak, and the regions are not in a relationship that should be merged. This merging relationship is described in the connection label column of the feature list shown in FIG. 35.

The label No. in the same list is a label value, and the area 1 centroid is the area and centroid of the area of that label value. This area and center of gravity are obtained during the labeling process described above.

A circumscribing rectangle is a rectangle that encloses all pixels within the region of the label value.

A circumscribed rectangle has four vertices that are intersections between sides,
It is specified by the coordinates tL (x, y) of the top left vertex and the coordinates bR (x, y) of the bottom right vertex. Each coordinate is determined during the boundary exploration process described above. Further, the label No. written in the connection label column indicates the label value of the area that should be merged with the area of that label value. If there is an area to be merged with the area described in this column, connection labels are further connected by a pointer as shown in the figure.

Further, each feature amount described in the same list is for each area shown in FIG. 36. Each area has a label value IIl~
p4 is attached, and 01 to G4 indicate the center of gravity of each region. Also, a circumscribed rectangle is drawn in each area. The point of contact between the upper side of this rectangle and the area is the coordinate s t
(x, y), and the region starting point tL (x
, y) by horizontal operation. The coordinate st is used in the method described later.

Since the region with the label value p1 and the region with the label value 112 are in a relationship that should be merged, the label No. of the feature value list,
A label value p2 is written in the connection label field of the row of ρ1. Further, since the area with the label value g1 is in a relationship to be merged with the area with the label value Ω3 and the area with the label value g4, the connected labels of the label values g3 and g4 are connected by a pointer. Furthermore, the connection relationships between the areas are described in the same way in the connection label columns of the rows of label No. f12 to Ω4.

Next, the area boundary exploration method will be explained.

First, draw a rectangle that circumscribes the target area, and the coordinates 5t (x) where this circumscribed rectangle and the area touch.

Find the search start pixel located at y). Then, eight pixels located in the vicinity of this search start pixel are investigated.

8. Each reference pixel located in the vicinity is given a position number as shown in FIG. That is, the number located above the pixel of interest is set to 0, and the numbers are assigned in clockwise order. The coordinates of the pixel to be searched next to the search start pixel are shown in Figure 38 (a
), (b) using the pixel reference table.

The table shown in (a) of the same figure is a table that is referred to when performing a pixel search clockwise (clockwise).
The table shown in FIG. 4B is a table that is referred to when performing a pixel search counterclockwise (counterclockwise). The numbers from O to 7 in parentheses () in the table are the third
This corresponds to the position No. of the reference pixel in FIG.

The next pixel to be searched is the reference pixel position No. corresponding to the current pixel position relative to the previous pixel position.

Find the row number of the table that matches this position number,
It is determined by referring to the column No. A method for determining search pixels using a pixel reference table will be specifically described below. Assume that the boundary of the pixel area is formed as shown in FIG. 39. The hatched areas in the figure indicate pixels whose label values are not 0. The previously searched pixel is indicated by the symbol △, the currently searched pixel is indicated by the symbol O, and the next pixel to be searched is indicated by the symbol ◎.

The current pixel position O with respect to the previous pixel position △ is the 37th pixel position O.
The reference pixel position No. in the figure corresponds to 1.

If pixels near the boundary of the area are searched clockwise, the pixel reference table will refer to the table in FIG. 38(a). Reference pixel position No.

is 1, so the column number where the row number is 1, (7°0
．． See 1.2, 3, 4, 5.61. In addition, line No.
, the top row is O, and the numbers 1 to 7 are in order.

Further, column No. indicates all the numbers of the pixels to be referenced, and by referring to the eight neighboring pixels of the pixel in this order, it is possible to avoid modulo calculation in normal address calculation.

In other words, since the column number starts from 7, the next pixel to be searched is the pixel located diagonally to the upper left of the current position ◎
become. Since the pixel ◎ is in the background area and the label value is 0, the next pixel is searched according to the column number. In other words,
Next reference pixel position number.

Since is 0, a pixel located above the current position O is searched. Since this pixel is also in the background area, the column No.
, a pixel whose reference pixel position No. is 1 is searched for.

Since this pixel has a label value and is not 0, the reference for the next pixel to be searched is set to the pixel at this position No. 1, and the eight neighboring pixels are searched in the same manner as above. and,
Perform the above processing for each pixel along the region boundary,
Similar processing is repeated for each pixel until returning to the search start pixel.

In addition, in this search process, by performing processing according to the following rules, the perimeter o of the own area of label value i
common boundary length c between bi and adjacent areas of label value j
blj can be found.

In other words, if the label value j exists in four neighboring areas on the upper, lower, left, and right sides of the current position, 1 is added to the counter that counts the common boundary length cb1j. At the same time, 1 is added to a counter that counts its own perimeter obi. Also,
If there is no pixel with the same label value as its own label value i in the four vicinity of the current position, 1 is added to its own perimeter counter.

Next, based on the obtained perimeter length and common boundary length, the degree of connectivity between each region is obtained in the same way as calculation in the inverted mask operation method. At the same time, a feature list is created in the same manner as described above. The connection label of this feature list describes the connection relationship between each area, and the merging relationship of multiple areas is determined.

In other words, even with this method, the merging relationship of each area can be determined without changing the label of each area. Although the above boundary search was performed clockwise, if the boundary search is to be performed counterclockwise, it can be performed clockwise or counterclockwise by using the counterclockwise pixel reference table shown in FIG. 38(b). Processing can be performed in a similar manner.

Using a pixel lookup table in this way allows for fast boundary exploration. In other words, it is not necessary to investigate all eight pixels in the vicinity of the pixel serving as the search reference. Further, it is only necessary to search for pixels near the boundary of the area, and there is no need to search for all pixels within the area. Therefore, processing time is reduced.

According to each of the methods described above, it is possible to determine the merging relationship between the divided regions without changing the labeling and scanning as in the conventional method. In other words, the road candidate areas obtained by the ``driving course extraction method using iterative threshold processing using template images'' and the ``extraction method of shadows and high-brightness parts on the driving course that focuses on differences in brightness'' are used. The obtained low-luminance and high-luminance areas are combined to obtain a realistic driving course area.

Next, the "means for determining the travelable range" will be explained. This method consists of ``a driving course extraction method using iterative threshold processing using a template image,'' ``a threshold setting method based on the shape of the feature histogram in iterative threshold processing,'' and ``a method that focuses on differences in brightness. The final drivable range, that is, the road edge coordinates, is determined from the driving course image obtained by the ``extraction means for extracting shadows and high-luminance portions on the driving course''.

FIG. 40 is a flowchart showing the overall processing flow of the driving course recognition system according to the present method.

First, image information of a driving course is captured by a color camera input device, and the captured R, G, and B images are converted into images of brightness I and saturation S as described above. Based on each converted image, a histogram having lightness I as a feature and a histogram having saturation S as a feature are created in the color processing device as described above (step 400
1). Next, based on each of the created histograms, road candidate regions are extracted using a "driving course extraction method based on iterative threshold processing using a template image" and small regions are removed (step 4002).

Next, using this method, the point sequence of the road edge of the extracted road candidate area is evaluated (step 4003).

Then, based on the evaluation result of this road edge point sequence, it is determined whether the extracted road candidate area is a monotonous road or not (step 4).
004). In addition, if the road is monotonous like a road,
The relationships between the obtained plurality of point sequences are compared and evaluated (step 4005). Then, this evaluation result is finally output (step 4006). After this, it is assumed that the point sequence for the currently input image has been obtained, and processing is executed for the next image.

In addition, if the judgment result in step 4004 is that the road is not a monotonous road with low road-likeness, the "extraction means for extracting shadows and high-brightness parts on the driving course that focuses on differences in brightness" is used to detect low-brightness areas. Extract this (step 4007)
. Then, the obtained low-luminance area and the road candidate area are merged, and the road edge point sequence is evaluated using this method (step 40
08).

Based on the evaluation result of this road edge point sequence, it is determined whether the extracted road candidate area is a monotonous road or not (step 4009).
. If the path is monotonous, the process from step 4005 onwards is executed, and the process moves on to the next image.

In addition, if the road is not monotonous, next, if there is a high brightness area, it is extracted using "a means for extracting shadows and high brightness parts on the driving course that focuses on differences in brightness" (step 4).
010). Then, the obtained high-luminance region and the road candidate region are merged, and the road edge point sequence is evaluated using this method (step 4011). Based on the evaluation result of this road edge point sequence, it is determined whether the extracted road candidate area is a monotonous road (step 4012). If the road is monotonous, step 4
The processes after 005 are executed, and the process moves on to the next image. If the road is not monotonous, the low-luminance area and the high-luminance area are merged into the road candidate area, and the road edge point sequence obtained from the merged area of the three areas is evaluated (step 401
B).

Thereafter, the process of moving to step 4005 is executed, and the process moves to the next image.

Next, a method for obtaining the point sequence coordinates of the road edge in the road candidate area will be described below.

For example, assume that the image area shown in FIG. 41 is obtained. The upper left of the figure is the origin, the X coordinate is positive in the horizontal direction toward the right, and the X coordinate is positive in the vertical direction toward the bottom. There are notches in the road area shown in this image, and there are also scattered locations where image information could not be obtained due to noise or the like. It is assumed that labeling processing is performed on each pixel of the image shown in the same figure, and the resulting pixel area is shown as shown in FIG. 42 as a result of this processing. The shaded area in the figure is a pixel with a label value of a certain label value, rather than a label value of O. Note that pixels without diagonal lines are pixels located in a background area with a label value of 0. Also, a frame 4 surrounding 3 x 3 pixels
201 is a window W, and the number of diagonally lined pixels (pixels having non-zero pixel values) existing in the window W becomes the value of the histogram. The value of the histogram by window W in the case shown is 5. Also, although the illustrated window W is a 3×3 window, it is 5×
It may be a window of other size, such as 5 or the like.

First, the window W is scanned horizontally from the center of the image toward the left. Note that the window W is scanned from the center of the image only at the start of the process. The histogram value of the window W at each position is determined while moving. The window W is moved while monitoring this histogram value, and if histogram values below a predetermined threshold are continuously obtained, that is, if the density of diagonally lined pixels in the window W becomes low, the window W is It is determined that the vehicle is out of the road area. Then, the position of the first window W whose histogram value becomes less than or equal to a predetermined threshold value is set as the point sequence coordinate XL of the left road edge.

Next, the window W is horizontally scanned toward the right side, and when the histogram values within the window W are continuously below a predetermined threshold, the position of the window W where the histogram value first changes is set to the right side. Let the point sequence coordinates of the road edge be XR.

Next, the Y coordinate at which the window W is located is decreased by one, and the window W is moved above the image to shift the horizontal scanning position. Then, the center position of the road area is calculated using the formula (XR-
It is obtained by calculating XL)/2. This center position is set as the scanning start position of the window W, and the window W is scanned left and right. In this scanning as well, the histogram value within the window W is monitored in the same manner as above, and the position where the histogram value changes is taken as the coordinates of the road edge.

Thereafter, by repeating the same process until the Y coordinate of the horizontal scanning position reaches the horizon position, a series of point sequence coordinates of the left and right road edges can be obtained.

In addition, in scanning this window W, the histogram values obtained from the window W at the scanning start position are below the predetermined threshold from the beginning, and the histogram values obtained by scanning the window W to the left are continuous. If this is less than a predetermined threshold, the center position becomes a roadside seat. However, in this case, the scanning direction of the window W is changed to the right side. Then, when histogram values greater than or equal to a predetermined threshold are obtained consecutively, the position of the window W where a histogram value greater than or equal to the predetermined threshold is first obtained is set as the coordinates of the left road edge. . The coordinates of the right road edge can be found by scanning the window W further to the right and finding the position where the histogram value changes.

Next, since the point sequence coordinates of the road edge obtained in this way exist near the boundary of the area, the boundary line obtained by connecting each point is not uniformly smooth. For this reason, a certain 1
The angle formed by each point located before and after the point and this one point is calculated for all points in the point sequence, and this variance value is used as a standard for smoothness. Also, if the obtained angle is an acute angle, that point is removed from the calculation. After smoothing the point sequence coordinates of the left and right road edges in this manner, these point sequence coordinates are projectively transformed into real space. This projective transformation is performed based on the depression angle, height, and focal length of a color camera attached to the autonomous vehicle.

Furthermore, in the point sequence after projective transformation, the distance between the left and right point sequences, that is, the road width, is determined by setting the left and right point sequences as a set. Then, the number of points where this road width is narrower than the vehicle body width of the traveling vehicle is counted, and the number of points of this narrow width relative to the total number of points in the point sequence is calculated. If the ratio of the narrow width points to the total number of points is small, the narrow width points are not suitable as road edge points, so these points are removed.

In addition, when evaluating this road edge point sequence, the left and right pairs that have a small difference from the road width determined by the initially determined pair of left and right edges are counted. When the ratio of left and right sets with little difference to the total number of sets is large, that is, when the left and right road edge rows are parallel to some extent, the evaluation as road-likeness is highest, and the road is judged as a monotonous road. be done. If the path is determined to be monotonous in this way, steps 4003, 4008, 40 in the flowchart shown in FIG.
It is possible to obtain the road edge without performing the processing in step 11.

Next, a description will be given of means for correcting the point sequence at the left and right edges of the road when the autonomous vehicle approaches the road edge too much. Figures 43 (a) to (d) show the process in which an autonomous vehicle approaches the edge of the road. Figure 43 (a) shows a case where the autonomous vehicle is located in front of a curve with a sharp curvature. In this case, the color camera attached to the autonomous vehicle captures the left road edge and the right road edge R within its field of view.If the curvature of the road is steep,
The autonomous vehicle located in (a) of the figure moves to the position shown in (b) of the figure. In this case, the right road edge R disappears from the field of view of the camera.

Further, the autonomous vehicle moves to the position shown in FIG. 2(c), and the ratio of the off-road area to the template gradually increases. The illustrated shaded area indicates the area outside the road. Furthermore, the autonomous vehicle moves to the position shown in FIG.

In this case, the left road edge and the left boundary of the off-road area are captured in the field of view of the camera. Moreover, the ratio of the area outside the road to the template image becomes high, and the area outside the road is mistakenly recognized as the driving course. As a result,
The left road edge is mistakenly recognized as the right road edge R, and the left boundary of the off-road area is mistakenly recognized as the left road edge.

However, in the point sequence of the road edge obtained from sequentially captured image information, the point sequence on the right side in the previous image does not become the point sequence located extremely on the left side in the current image. Also,
Conversely, the point sequence on the left side in the previous image will not become the point sequence located extremely on the right side in the current image. Therefore,
The point sequence information obtained from the image information one processing cycle ago is stored and retained, and the distance between the point sequence obtained from the current image information and the point sequence obtained from the previous image information is calculated as described later. . For example, the distance between the point sequence of the right road edge found last time and the point sequence of the right road end found this time is far apart,
If the previous right side point sequence is close to the current left side point sequence, it is determined that the positions of the left and right road edges have been reversed. Then, swap the right and left sides of the road edge obtained this time.

This will be explained using the image shown in FIG. The image shown in FIG. 4C is the image obtained last time, and the point sequence at the left side of the road is expressed as Lif. The image in (d) of the figure is the image obtained this time, and the misidentified left road edge point sequence is Ll,
The misidentified right-side road edge point sequence is expressed as Ri. A determination as to whether or not the left and right road edge point sequences are reversed is made by determining which of the current point sequence Li and R1 the previous point sequence L1-1 is closer to. Previous point sequence L1-1
is close to the current point sequence R1, it is determined that the point sequences at the left and right road edges have been reversed.

- The proximity of this point sequence within the film is determined as follows. In other words, the left-hand point sequence obtained from this image is one group, the right-hand point sequence is another group, and it is determined which of these groups the previous left-hand or right-hand point sequence is closer to. Do by doing. The determination of which group it is closer to is determined by Mahalanobis' general distance, which will be explained in detail below. FIG. 44 is a diagram for explaining the Mahalanobis general distance.

The point of the left point sequence L1-1 (or right point sequence R1-1) according to the previous image information is (x, y), the point of the left point sequence Li according to the current image information is (x lj, y 1j), this time The points of the right-hand point sequence Ri based on the image information of (x 2j.

y2j). To determine which point (x, y) in the previous point sequence is closer to the current point sequence L1 or R1, calculate the Mahalanobis general distance for all points in the point sequence Ll-1 (or point sequence R1-1). Calculate as below for Furthermore, when the point sequence determined from the current image is a straight line, the distance to this straight line is calculated.

First, for each point on the point sequence L+, the average value μm1 of the X direction component. Average value of the X direction component μm2. Variance value σ11 of the X direction component. Variance value σ12 of the y-direction component. A covariance value σ112 and a correlation coefficient ρ1 in each of the x and y directions are determined. Here, ni (is the number of points located on the point sequence lj. Note that each point on the point sequence R4 can also be determined by a formula similar to the following formula.

μm1-Σxlj/nl, R12-Σy lj/
n 1σ112−Σ(xlj−μll) 2/(n
l −1) σ122−Σ(y lj−μm2) 2/
(nl −1)σ112−Σ(x lj−μm1)
(y lj - R12) / (nl -1) ρl - σ112 / σIIX σ12 Based on these values, the point (x) of the previous point sequence is calculated.

y) and the point (x lj, y lj) of the current point sequence L+
The general distance JIIDI'' of Ma. Lanobis between can be calculated using the following formula.

Di 2= ([(x-, CZII) / (711]
2+ [(y-, cz12) /cr12] 2-2ρ
1 [(x-μll) /σ11]X[(y-μm2
)/σ11] 1/(1-ρ1)2 Similarly, the point (x, y) of the previous point sequence and the point (x 2j) of the current point sequence R1
, y 2j)
2 can be found using the following formula.

D2 2-([(x - μ 21) / σ 21ko 2 + [(y-R22)/σ22] 2-2ρ2 [(
X-R21)/σ21]×[(y-μ22)
/σ22ko) l(1-ρ2)2 As a result of calculating each general distance based on each formula above, Dl2>D2
If 2 holds true, the previous point (x, y) is close to the current point sequence R1 of the right road edge. Also, Di2
〉If D22 does not hold, conversely, the previous point (x,
y) is close to the current point sequence L1 at the left road edge.

As described above, even if the road candidate region obtained as a result of the image segmentation process has an unclear outline due to noise, etc., the edge points of the road can be determined based on the changes in the histogram values of the window described above using this method. You can definitely get it.

This is because the present method searches for the end points of the region based on the region, and according to the present method, the left and right ends of the region can be clearly distinguished.

In addition, low-luminance areas found using the "extraction method for shadows and high-luminance areas on a driving course that focuses on differences in brightness" may be located outside the road if the driving course is dark overall. be. In such a case, the low-luminance area outside the road and the road area may be merged, and the shape of the outline of the road candidate area becomes complicated. However, by smoothing the obtained road edge point sequence data using this method, regions with complex contours can be evaluated in a global manner, and low-luminance regions such as those mentioned above are ignored. It is possible to obtain road edge information without any problems.

In addition, according to the ``driving course extraction method using iterative threshold processing using template images,'' when an autonomous vehicle approaches the left or right side of a road with a sharp curvature, the template may veer off the road. In some cases, the road may be considered a road.

In this case, the left and right sides of the road edge will be incorrectly recognized.

However, according to this method described above, even if the left and right edge data of the road is incorrectly recognized, this recognition is immediately corrected, and highly reliable road edge information can always be obtained.

Next, a description will be given of an "internal representation method for driving courses with various shapes," that is, a method for structuring and representing various point sequences of the obtained road edges.

This method uses an image of a driving course obtained by ``a means of extracting a driving course by iterative threshold processing using a template image'' and ``extracting shadows and high-brightness parts from the driving course by focusing on differences in brightness.'' Based on the image obtained by the "method for merging multiple regions" and the feature list obtained by the "method for merging multiple regions," a strafture of region boundary points is created while detecting the label displacement location of the image, and links in the forward and backward directions are created. The driving course is expressed inside the computer by adding boundary attributes and determining a valid point sequence for driving. By using the feature list representing the region and the description of the region created in "Means for merging multiple regions", the end points of the region can be quickly obtained, and the end points of the region can be quickly obtained, such as road edges, road shoulders, two branch roads, and merging roads. is represented by a structured list inside the computer.

FIG. 45 is a flowchart showing an outline of road structuring processing.

First, when initializing each piece of hardware such as a color camera input device, a list of region boundary point struc- tures in the X direction is created (step 4501).

Next, a change point in the X direction of the label value attached to each area is detected, a list in the X direction is created, and a link is made in the X direction of each boundary point (step 4502). Then, considering the connection label of the feature quantity list of each region, attributes are given to the left and right ends of the region, and the left and right ends of each region are divided (step 450B). Next, while adding attributes such as holes, which will be described later, to each boundary point,
The boundary points of each boundary point are associated and linked in the X direction of each boundary point (step 4504).Also, there are cases where road areas diverge or merge, and in each of these cases, the continuity of the boundaries of each area is Then, the attribute assigned to each boundary point is changed (step 4505). Thereafter, the starting position of the point sequence of the boundary points of each area is selected, and a valid point sequence is detected (step 4506).

Each of the above processes will be explained in detail below.

The process of step 4501 will be explained with reference to FIG. 46.

FIG. 5A is a diagram showing the principle of projective transformation of a road surface (assumed to be a horizontal surface) in real space onto an image. The road surface 4601 is divided into equal intervals (N), and the road area and each division line on the road are divided into the image 46 toward one point in the distance.
02 to obtain the viewport of the color camera. FIG. 4B is a diagram showing details of the image obtained in this manner. On the image there is a boundary line 4603 of the road area.
and the dividing line 4604 have been projectively transformed. In real space, the lane markings are equally spaced at intervals g, but in the image, the intervals become closer as you get farther from the road area. The y-coordinates at which each division line 4604 is located are determined, and an area boundary point structure 4605 in the X direction shown in FIG. 4(c) is created.

The six frames of this area boundary point struc- ture 4605 are the leftmost boundary line and the dividing line 46 of the road area boundary lines 4603.
This corresponds to each boundary point located at the intersection with 04. The mutual connection relationship of the six frames is expressed by pointers indicated by arrows in the figure.

In each of these frames, the characteristics of each boundary point are described as a list 4606 shown in FIG. 4(d).

First, the six values of the determined y coordinate are written in a list 4606, and a list in the X direction is created. In addition to the above y-coordinate, this list 4606 describes each feature that each boundary point has, which will be described later.

In other words, attribute 1 representing the X coordinate 1 point sequence start point iD, attribute 22 representing the distance between adjacent points in real space, pointer to the right neighbor representing the connection relationship between the boundary point of interest and the boundary point to the right, pointer to the left neighbor A pointer to the left that represents the connection relationship with the boundary point, a pointer to the front that represents the connection relationship with the boundary point in front of the screen in the A pointer is written.

Next, the process of step 4502 will be explained with reference to FIG. 47.

An image 4701 shown in FIG. 4A is an image obtained by performing labeling processing on a captured image. Through this labeling process, each region constituting the road region is assigned a label value. Among the intersections between each division line and the area boundary, the intersection marked with an O mark is a label displacement location where the label value changes. First, step 4501
The image is scanned in the horizontal direction, that is, in the X direction at each y-coordinate position of the list obtained by the process. In this scan, if you move from a background area whose label value is 0 to a road area whose label value is not 0, or if you move from a road area whose label value is not 0 to a background area whose label value is 0, Create a new region boundary point structure.

This region boundary point struc- ture is shown in FIG. 4(b).

As shown in the figure, a new frame is provided in the X direction corresponding to each boundary point located at the intersection of the boundary of each area and the division line. These six new frames are connected to the six frames of the y-direction area boundary point structure already created in the previous step.

This connection relationship is expressed by a pointer represented by an arrow in the figure, and the six frames are linked in the horizontal direction by this pointer.

Next, the process of step 4503 will be explained with reference to FIG.

It is determined whether each set of points in the horizontal direction of the area boundary point structure created in step 4502 exists in the same area. A determination as to whether the regions are the same is made based on the feature quantity list obtained in the above-mentioned "method for merging multiple regions."

When adjacent points exist in the same area, the boundary point located on the left side is treated as a left edge, and the boundary point located on the right side is treated as a right edge, and this attribute is described in each list. This attribute allows the left and right ends of the area to be distinguished. Furthermore, the distance in real space from this left edge to the right edge is calculated and similarly written in each list. This distance calculation involves performing inverse projective transformation processing, which is the inverse of the projective transformation described above.
It is necessary to find the actual road area from the image.

The edge identification method for the image shown in FIG. 48(a) will be specifically explained. The road area 4801 is expressed by merging three areas A, B, and C. Here, the connection labels of the above-mentioned feature quantity list are shown as shown in FIG. 2(b). In other words, the connection label column for area A contains area C.
is written, nothing is written in the connection label column of area B, and area A is written in the connection label column of area C. Therefore, it is determined that area A and area C are the same area to be merged, and it is also determined that area B is a different area from the area formed by area A and area B.

Therefore, when the image is scanned horizontally along the dividing line 4802, the boundary points a, b, c marked with O,
d is determined as the label displacement location.

Therefore, boundary point a and boundary point S exist as points on the same area, boundary point a is given the attribute of left edge, boundary point A is given the attribute of right edge, and is described in the list of area boundary point structure. Ru. In addition, boundary point C and boundary point d exist in the same area because their label values are the same, and boundary point C is given the attribute of left edge, boundary point d is given the right edge, and the area boundary point struc- ture is described in the list.

Next, the process of step 4504 will be explained with reference to FIGS. 49 and 50.

In the image shown in FIG. 49(a), each boundary point located at the intersection of the left boundary line and each division line is a j , a
I”l, the boundary points located at the intersections of the right boundary line and each division line are bi, bl+1. Here, the i-th and i+1-th boundary points in the y direction of the area boundary point struc- ture are the same. It is determined whether or not it exists in the area, and if it is the same area, it is linked before and after.In other words, the boundary points at and a tr
i is in the same area, and the boundary points bj and b
It is determined whether or not the ``ill'' and ``ill'' are in the same area. In the case shown in the figure, each boundary point is in the same area, so the figure (b)
In the area boundary point structure shown in , each boundary point located before and after the screen is linked by a pointer shown by a thick line.

Furthermore, if there is an internal point existing between the left edge and the right edge, the boundary of the area is traced as shown in FIG. The figure (a) shows a road area 50 to be processed.
This is an image in which white 15002, which is a lane marking, is drawn in the center of 01.

The label value of this white line 5002 is 0. Here, when scanning the image horizontally along each dividing line, the white line 500
In the second part, the internal point E shown in FIG. 11(b) 11.
E 1i+1. E21. E 2i+1 is obtained.

Therefore, the boundaries of the region are tracked to explore the relationships between each internal point. If this boundary is traced along the boundary line of the white line 5002 from the interior point Elf, the interior point E 1141 is reached. Also, when tracing the boundary from the interior point E2j, the interior point E2i+
reach l. In this way, adjacent internal points E11.
When E 2i is continuous to the next internal point E t++t, E 21+1 on the dividing line located above, the internal point Elf located on the left side is the internal point left edge, and the internal point E2+ located on the right side is Give an attribute of internal point right edge.

Further, when tracing the boundary and reaching an adjacent internal point without being continuous with an internal point on the upper division line, each adjacent internal point is given the attribute of hole. In other words, the interior point E
Tracing the boundary from 11+1, the adjacent interior point E 2
reaches i+1. In this case, each internal point E II+l,
Give E 2i+1 the attribute of hole.

The above processing is performed along each y-coordinate in the list until this y-coordinate reaches the horizon position 5003. At this time, the distance in real space from the left edge of the interior point to the right edge of the interior point is calculated. This calculation is obtained by subtracting the X-coordinate value of the left edge point in real space from the X-coordinate value of the right edge point in real space.

Next, the process of step 4505 will be explained with reference to FIG. 51.

In the method of assigning attributes by processing each step above,
If a road branches into a Y-shape or merges, correct attributes will not be assigned. In other words, the association of each boundary point in the front-back direction may not be performed correctly. For this reason,
In such a case, in this step, the boundary is tracked and the boundary evaluation is corrected.

For example, suppose that the image shown in FIG. 51(a) is obtained through the processing of each step up to this point.

A road area 5101 in this image branches into a Y-shape, and is divided into a main road 5102 and a branch road 5103. Each dividing line (i, i+1
．． i+2. ) is image scanned, and the image is scanned from the left side of the screen to the right side at each dividing line. Through this scanning, the boundary point located at the intersection of the leftmost boundary line and each division line is given the symbol a, and the boundary point located at the intersection of the boundary line located one place to the right of the leftmost boundary line and the division line is assigned the symbol a. The boundary point located at the intersection of the boundary line two positions to the right of the leftmost boundary line and the division line is labeled C. As a result, the region boundary point struc- ture shown in FIG. 2(b) is obtained.

As can be seen from the diagram, the association of each boundary point by this region boundary point struc- ture does not match the actual road boundary, and each boundary point is not given a correct attribute.

Therefore, the boundary of the area is traced from the internal point, and if the next point is a boundary point that exists at the left or right edge of the area, this internal point and the next point are determined to be connected, The links at each of these points will be changed. In the case of the figure, the boundary of the area is traced from the internal point CI+4. When traced, the next point is a1+5, and since this point is the edge of the left area, the link attachment is changed. In other words, the attribute given to each boundary point after a1+5 is the same attribute of C as the internal point c i+4. This link was added in the same figure (
The region boundary point structure shown in c) is obtained. This strafture matches the boundaries of the actual road.

Figures (d) and (e) are diagrams showing the state of the area boundary before and after the link replacement process using vector representations. In other words, FIG. 4(d) shows the vector state before the link assignment is changed, and the connection relationship of each boundary line is expressed by a vector based on the area boundary point struc- ture shown in FIG. 4(b). The vectors to which attribute a is attached and the vectors to which attribute C is attached do not correspond to actual road boundaries. Figure (e) shows the vector state after the link replacement process is performed, and the connection relationship between each boundary line is expressed by a vector based on the corrected region boundary point structure in Figure (c). It is. The terminal portion of the vector to which the attribute a has been attached is modified as described above, and the attribute C has been assigned. Therefore, the boundaries of the road area represented by the vectors a, b, and c have shapes that match reality.

Figures (f) and (g) are diagrams in which the state of area boundaries when roads merge are expressed by vectors in the same way as above. FIG. 3(f) is a vector representation of the area boundary before the link is added. The vector with the attribute d blocks the entrance of the confluence path formed by the vectors with the attributes e and f. Same figure (g)
is a vector representation of the area boundary after the same linking and replacing process as described above. The starting end portion of the vector to which the attribute d has been attached is modified to the attribute f by the link reassignment processing, and the area boundary matches the actual confluence route.

Next, the process of step 4506 will be explained with reference to FIG. 52.

For example, assume the road image shown in FIG. This road image includes a road area 5201 and a road area 52.
On the left side of 01 is a road shoulder 5203 separated by a white line 5202.
There is. Boundary points are attached to the intersections between the boundary lines of each region of this road image and the lane markings, and the point sequence of the road edge and road shoulder edge shown in FIG. 4(b) is obtained. Note that points that overlap the edges of the image are excluded from the point sequence. The links of the region boundary point struc- tures in the y direction obtained by the operations in each step above are traced from the part corresponding to the bottom of the image to the top. If there are more than a predetermined number of links of the area boundary point struc- ture, an attribute of a starting point of a point sequence is attached to a location corresponding to the lower part of the image of the links of this area boundary point struc- ture. The starting point of the illustrated sequence of points is indicated by a Δ symbol.

Among the struttles with the attribute of starting point, a set of struttles whose distance in real space from the left edge to the right edge of the area boundary is equal to or greater than the vehicle width of the traveling vehicle is defined as a road edge. In the case of the same figure (b), point sequence 5204 and point sequence 5
The struture set corresponding to 205 becomes the road edge. In addition, there is a set of straftures next to the road edge, and the distance between this strafture group and the road edge is narrower than the width of the straffure group representing the road edge, and the distance between the straffure group and the other road edge is is wider than the vehicle width, and if the actual distance between struture groups is approximately 20 cm and continuous, then this struture group is assumed to form a white line. The area between the strafture group and one road edge is considered to be a shoulder separated by a white line.

In the sequence of points in FIG. 2B, a white line is formed by a set of strutures corresponding to the sequence of points 52o6 and 5207, and the area between this white line and the road edge 52o4 is considered to be the shoulder.

By the way, the above-mentioned "means for determining the drivable range" recognizes the driving course based on the area, and is based on the premise that the road area covered as a result of image processing is one area. According to this means, it is possible to identify the road edge of a monotonous road or a branch road. However, when an area is divided by a white line, especially into a road and a road shoulder, it is difficult to distinguish between them. This is because the means scans the window from inside the area to search for area boundaries, and therefore cannot identify the edge of the road and the edge of the road at the same time.

However, as described above, it is possible to know the boundaries of an area by scanning the label image horizontally from the bottom to the top of the image, tracking the boundaries of local areas, and finding the displacement location of the label value of each area. I can do it.

At the same time, the relationship between regions is updated in the strafture, and the boundary points of the regions are structured, so that the road region is represented in a structured manner inside the computer.

Therefore, according to this "internal representation method for driving courses with various shapes", the boundary edge coordinates of multiple areas such as road shoulders and driving lanes can be obtained, and the area boundaries can be easily detected.
The target area is not limited to a single area as in the case of ``Means for Determining Drivable Range''. Furthermore, it is also possible to identify road shoulders divided by the above-mentioned white lines.

In addition, it requires less memory capacity than conventional domain boundary methods, and evaluates the possible travel range of a driving course that is sufficient for an autonomous vehicle to drive automatically by performing sequential boundary exploration and polygon approximation. This can be obtained without any processing. Furthermore, even if the driving course is not a monotonous road, and the roads are branching or merging, the shapes of these branching roads and merging roads can be expressed uniformly, and the boundaries of the driving course can be accurately determined. It is possible to identify

As explained above, according to the "internal representation method for travel courses with various shapes", the connection relationships between the boundary points of each area are expressed in a structured strafcha using the attributes described in the list, and from this strafcha. The boundaries of the region of interest are identified. Therefore, since the number of point sequences that can be determined is large as in the conventional method, the process of approximating the boundary of the target area with a polygon and thinning out unnecessary points is no longer necessary. Further, it is no longer necessary to sequentially track all the boundary pixels of each area one by one as in the conventional method, and the target area is identified by processing only predetermined boundary pixels. Therefore, the processing time is shortened, and an area identification method suitable for autonomous driving of autonomous vehicles is provided.

〔Effect of the invention〕

As described above, according to the present invention, it is possible to extract a target area from a color image at low cost, at high speed, and accurately.

Further, the conversion process from RGB data to ISH data is performed only when storing ISH data in each storage element, and thereafter the ISH data can be directly read from each storage element. Therefore, when ISH data is needed, it is only necessary to read out the data in the storage element, and there is no need to convert RGB data to ISH data each time as in the past, allowing for high-speed calculations. processing becomes possible.

Furthermore, the relationship between each image divided into regions based on the threshold value and the driving road region is determined by calculating the degree of overlap with the template image. Therefore, even if there are no white lines or landmarks on the road, it is possible to detect the road area. Further, it is possible to detect the driving road area regardless of the state of the light source illuminating the driving road or unclear white lines. Furthermore, even if there is trash or a shadow on the road, accurate information on the road can be obtained. In addition, even if the boundary between the driving path area and the background area is divided by grass or dirt, or if the brightness of the driving path changes from moment to moment depending on the weather or the position of the sun, the correct driving path can be maintained. You can find the area. Moreover, processing using these template images can be efficiently realized without requiring any special equipment, and processing costs can be suppressed.

Further, similar color areas having lower luminance than the driving road area are individually extracted as corresponding to parts such as shadows formed on the driving road. Similarly, areas of similar colors that are brighter than the driving path area are individually extracted as corresponding to areas such as sunlight illuminated on the driving path. This eliminates the need to measure the color distribution of a limited driving road area in advance and compare it with the input image, as was the case in the past. It is now possible to extract low-brightness parts and high-brightness parts individually and at high speed even when the road is wide.

Furthermore, by using a labeling processor, labeling processing can also be performed on multivalued input images. Furthermore, by performing labeling using runs, the circuit scale of the labeling processing device can be reduced. Furthermore, by using the label matching memory, label integration becomes unnecessary, the label integration time becomes zero, and the labeling processing time is shortened. In addition, by configuring the label matching memory using a labeling memory that can simultaneously rewrite information, real-time processing becomes possible.

Furthermore, by using the feature extraction processor, discontinuous temporary labels generated in the primary labeling can be made into continuous final labels. Furthermore, since the final labels are continuous and the data is compressed, the execution time for subsequent feature quantity extraction is shortened. Furthermore, the feature extraction processor can simultaneously calculate the area of the same label region and the center of gravity of the same label region during secondary labeling.

Until now, this calculation has been performed by performing one more scan after two scans of labeling.

Furthermore, by performing labeling from the lower right to the upper left of the screen, the road area can be reliably extracted even if the temporary label overflows. Further, by configuring the label memory in which output label values are stored using a labeling memory in which information can be simultaneously rewritten, one-scan labeling becomes possible.

In addition, it is no longer necessary to change label values, which takes a long processing time to merge regions, and it is now possible to merge multiple regions in a short time. Therefore, high-speed processing required for self-sustaining automobiles is possible, and processing costs are reduced.

In addition, the range of the target area is determined based on the number of pixels existing in the window, and by scanning the window horizontally from side to side over this target area and examining changes in the number of pixels within the window, the range of the target area and the background can be determined. The boundary with the matching area is identified.

Therefore, even if the outline of the area is unclear or has a complex shape due to notches, etc., the left and right edges of the boundary can be reliably detected in order to search the boundary based on the area. You can get it. In addition, by smoothing the point sequence at the left and right ends of the obtained boundary, a contour with a complex shape can be judged from a global perspective, and it is possible to obtain a point sequence at the left and right ends sufficient for driving.

Furthermore, even if the left and right edges of the boundary are temporarily incorrect due to vehicle movement, this data is immediately corrected, making it possible to obtain highly reliable information. Furthermore, since the point sequence is evaluated in this way and the process moves to the next image processing when a valid point sequence is obtained, processing efficiency is improved.

[Brief explanation of drawings]

FIG. 1 is a block diagram showing a schematic configuration of the entire color image processing apparatus according to an embodiment of the present invention, FIG. 2 is a flowchart showing a schematic processing flow of this embodiment, and FIG. Block diagram illustrating the algorithm of ISH conversion processing in ``ISH conversion processing of color images'', Part 4
5, 6, and 7 are diagrams for explaining a processing example of this ISH conversion process in comparison with a conventional processing example, and FIG. A flowchart showing an outline of the preprocessing of a color image in "threshold setting means based on the shape of the histogram", Fig. 9 is a diagram for explaining the state of peaks and valleys in this preprocessing, and Fig. Figure 11 is a flowchart showing an overview of the travel route recognition process, and Figure 12 is a diagram for explaining the transition of table values in preprocessing.
FIG. 13 is a flowchart showing the details of the driving route extraction process in the driving course extraction means by repeated threshold processing using a template image, and FIG. Figure 14 is a diagram for explaining the outline of the processing for various input images in the ``Method for extracting shadows and high-luminance parts from a driving course that focuses on differences in brightness.'' A graph showing an example of a brightness histogram to explain the luminance part, Fig. 16 is a labeling board configuration diagram of the "labeling processing device", Fig. 17 is a general flowchart of labeling processing, and Fig. 18 is a multi-value input labeling method. Figure 19 is a diagram to explain the labeling method using runs, Figure 20 is a diagram showing the window used in image scanning for labeling processing, and Figure 21 is a diagram of the labeling processor KLP. 22, FIG. 23, FIG. 24, FIG. 25, FIG. 26, and FIG. 27 are flowcharts showing window processing performed in image scanning of labeling processing, and FIG. 28 is a flowchart showing the labeling memory KLM. Configuration diagram,
FIG. 29 is a diagram for explaining the process of matching finally assigned label values, and FIG. 30 is a diagram for explaining the process of matching the label values finally assigned.
A configuration diagram of the LC, FIG. 31 is a diagram for explaining the 1-scan labeling method, and FIG. 32 is a diagram showing an inverted mask used to search the label area in the "merging means of multiple areas" , FIG. 33 is a diagram showing an example of divided regions used to explain the area boundary exploration using the inverted mask shown in FIG. 32, and FIG. A diagram showing an inter-label boundary length madrisk in which the determined boundary length between label regions and the perimeter of the region are stored, and FIG. 35 is a feature value list that describes the connection relationship between each region and the features possessed by each label region. A diagram showing
Fig. 36 is an example of a label region having the features described in the feature list shown in Fig. 35, and Fig. 37 is a reference pixel position attached near the pixel of interest in the boundary search method. No. , FIG. 38 is a diagram showing a pixel reference table used in the boundary search method, and FIG. 39 is a diagram showing a pixel area used to explain how to use the pixel reference table shown in FIG. 38. FIG. 40 is a flowchart showing the processing flow of the driving course recognition system in the "means for determining the drivable range", and FIG. 41 shows the movement of the window in the captured image used in the explanation of this means. Figure 42 is a diagram for explaining the window and histogram values in the target area, Figure 43 is a diagram for explaining the window and histogram values in the target area.
The figure is a diagram to explain the reversal of road edge recognition that occurs when a vehicle approaches the road edge too closely on a curve with a tight curvature. Figure 44 shows the reversal of road edge recognition shown in Figure 43. Diagram 45 for explaining the method of correcting the relationship of the point sequence obtained to prevent the problem using Mahalanobis' general distance
The figure is a flowchart outlining the process of structuring a road area in the ``internal representation method for driving courses with various shapes.''
Fig. 46 is a diagram for explaining the y-direction list and the area boundary point struc- ture, Fig. 47 is a diagram for explaining the label displacement location and the area boundary point struc- ture, and Fig. 48 is a diagram for explaining the attributes of the area boundary. FIG. 49 is a diagram for explaining link processing in the front and back direction of area boundary points, and FIG.
The figure is a diagram to explain the attributes of internal point edges and holes, Figure 51 is a diagram to explain the process of replacing attributes once assigned, and Figure 52 is a diagram to explain the detection of road edges and road shoulder edges. This is a diagram for 101... Color camera, 102... ISH conversion unit, 103... Color processing unit, 104... Labeling hardware, 105... CPU processing unit.

Claims (1)

[Scope of Claims] 1. An imaging device that captures an original image as RGB information, and a system that uses the RGB information captured by the imaging device to determine brightness, saturation,
an ISH conversion unit that converts into hue ISH information;
a color processing unit that extracts a target area based on the IS information of brightness and saturation converted by the conversion unit; and a labeling unit that labels each area forming the target area extracted by the color processing unit. , merging means for determining the connection relationship between the regions labeled by the labeling unit and performing merging processing between the regions, and region boundary identification means for determining the boundary edges of the merged regions. , a color image processing device characterized by identifying the shape of an imaged target area. 2. An original image memory in which captured original image data is stored, and R of RGB data included in the original image data.
an R image memory that stores data; a G image memory that stores G data of the RGB data; and a G image memory that stores G data of the RGB data;
a B image memory that stores B data of the data; a first storage element that stores brightness data converted from the RGB data stored in the R image memory, the G image memory, and the B image memory; , the R image memory;
a second storage element in which saturation data converted from the RGB data stored in the G image memory and the B image memory is stored; a third storage element in which hue data converted from the RGB data is stored;
An ISH conversion device for color image processing, comprising: an average value filter that averages the hue data output from the third storage element. 3. Extract RGB data from the captured original image data, convert the RGB data into brightness data, chroma data, and hue data, and store the brightness data, chroma data, and hue data in separate storage elements. The brightness data and saturation data are read out from each storage element and directly subjected to image processing, and the hue data is read out from the storage element and passed through an average value filter before being subjected to image processing. An ISH conversion method in color image processing, characterized in that: 4. Extract RGB data from the captured original image data, convert the RGB data to brightness data, create a feature histogram from the brightness data, and set area segmentation thresholds based on the feature histogram. Then, the original image is binarized using the area segmentation threshold to create segmented images, and the target area is binarized into a predetermined area assumed to be at the image position where the image is captured and other areas. calculating the degree of overlap between the template image and the divided image;
A method for extracting a target region, comprising: determining that a target region is included in the divided images having a high degree of overlap, and extracting the target region from a background region. 5. Convert RGB data to saturation data, create a feature histogram from the saturation data, set a region segmentation threshold based on the feature histogram, and divide the original image using the region segmentation threshold. A divided image is created by dividing the area by binarization, the degree of overlap between the template image and the divided image is calculated, and it is determined that the target area is included in the divided image with the higher degree of overlap. 5. The target area extraction method according to claim 4, further comprising extracting a target area based on the target area and determining a more realistic target area based on the target area and the target area extracted from the brightness data. 6. The setting of the region segmentation threshold based on the feature histogram is characterized in that the threshold obtained by Otsu's discriminant analysis method is corrected by the valley candidate value obtained based on the shape of the feature histogram. 5. The target area extraction method according to claim 4. 7. The setting of the region segmentation threshold based on the feature histogram is characterized in that the threshold obtained by Otsu's discriminant analysis method is corrected by the valley candidate value obtained based on the shape of the feature histogram. 6. The target area extraction method according to claim 5. 8. If there is a valley candidate value found based on the shape of the feature value histogram in a divided image with a high degree of overlap with the template image, the divided image is further divided into regions according to the candidate value, and the divided image is divided into regions according to the candidate value. 7. The target area extraction method according to claim 6, further comprising repeating the area division until the candidate value is no longer present in the divided image. 9. If there is a valley candidate value determined based on the shape of the feature value histogram in the divided image with a high degree of overlap with the template image, the divided image is further divided into regions according to the candidate value, and the divided image is divided into regions according to the candidate value. 8. The target region extraction method according to claim 7, wherein region division is repeated until the candidate value is no longer present in the divided image. 10. If there is a region darker or brighter than the target region extracted from the brightness data, the dark or bright region is binarized based on the valley candidate value obtained based on the shape of the feature value histogram, and the target region is extracted from the brightness data. The bright area or the dark area is extracted based on the target area image obtained from the binarized brightness image and the chroma image. Target area extraction method. 11. The target area extraction method according to claim 10, characterized in that the extracted bright area or dark area is merged with the target area determined from the brightness image and the chroma image. 12. An input memory that stores multivalued image information obtained by adding together a plurality of binary input images, a labeling processor that labels each pixel stored in the input memory according to the pixel value, and the labeling processor. What is claimed is: 1. A labeling processing device comprising: a label memory for storing output label values of each pixel assigned by the method, and capable of performing labeling processing even on multivalued input images. 13. The labeling processor compares the connection relationship between the pixel of interest and the adjacent pixels based on the label value with a first comparison circuit that compares the label values of the pixel of interest and each pixel adjacent to the pixel of interest. Claim characterized in that it is configured to include a second comparison circuit and a selector circuit that determines a label value of a pixel of interest based on comparison results of the first comparison circuit and the second comparison circuit. 12
Labeling device as described. 14. A label matching memory is provided to store the temporary label value of each pixel attached by the primary scanning by the labeling processor, and the temporary label values stored in the label matching memory are arranged in the secondary scanning by the labeling processor to form a label. Claim 12 characterized in that the image is output after being stored in memory as an output label value.
Or the labeling device according to claim 13. 15. The label matching memory includes a storage circuit that stores information, a comparison circuit that compares the information stored in the storage circuit and input information from the outside and outputs a signal when the information matches, and an address a decoder circuit that inputs a signal, decodes and outputs the signal; and outputs a signal to an enable terminal of the storage circuit when a valid signal is output from any one of the comparator circuit and the decoder circuit; 15. The labeling device according to claim 14, characterized in that the labeling device comprises a labeling memory having a logic circuit as one unit of the configuration, which makes the information stored in the storage circuit into a rewritable state, and eliminates the need for label integration. . 16. Claim 12, Claim 13, Claim 14, or Claim 1, characterized in that each pixel is scanned by the labeling processor from the bottom right to the top left of the screen of the input image.
5. The labeling device according to 5. 17. Equipped with a line buffer memory configured by a labeling memory to store image information for one scan, and when the pixel area of the input image is formed in a step-like manner,
The labeling processor divides the pixel area of the input image into runs for each row, sets a flag in the line buffer memory as long as pixels with the same label value continue during scanning of the run, and sets a flag in the line buffer memory at the end of the same label pixel or at the end of the same label pixel. 16. The labeling apparatus according to claim 12, 13, 14, or 15, wherein a label is attached to each flagged pixel at the same time when the last pixel of a run is scanned. 18. A first counter circuit that outputs count pulses generated in synchronization with a predetermined clock as an address to the label matching memory, and a second counter circuit that outputs count pulses generated in synchronization with the clock as data to the label matching memory. A counter circuit compares each value of the address outputted by the first counter circuit and the data outputted by the second counter circuit, and if the respective values are different, the count-up of the second counter circuit is stopped. Claim 1, further comprising: a comparison circuit, which converts temporary label values of discontinuous values temporarily stored in a label matching memory into continuous values through primary scanning by a labeling processor.
4. The labeling device according to claim 4, claim 15, or claim 17. 19. A counter circuit that counts up by one each time a pixel with the same label value is input, and a first adder that adds the address value of the X-direction address of the pixel each time a pixel with the same label value is input. and a second adder that adds the address value of the Y-direction address of the pixel each time a pixel with the same label value is input, and when performing secondary labeling by the labeling processor, the area of the same label area , the total value of the X-direction addresses of the same label area and the total value of the Y-direction addresses of the same label area are determined.
18. The labeling device according to claim 5 or claim 17. 20. The labeling apparatus according to claim 12 or 13, wherein the label memory for storing the output label value is constituted by a labeling memory, and the labeling is performed by performing only one scan of pixels by a labeling processor. 21. A mask is scanned for the pixels in each area that has been divided into images and labeled, and the labels of the same labeled area are determined based on the connection relationship between the label value of the pixel of interest appearing in the mask and the label value of the pixel adjacent to the pixel of interest. Determine the perimeter and the common boundary length between the regions adjacent to the same labeled region, determine the degree of connectivity between the adjacent regions based on the ratio of the perimeter and the common boundary length, and store this degree of connectivity in memory. A method for merging multiple areas, characterized by describing and determining connections between each area. 22. A pixel reference table is created in advance in which each pixel located in the vicinity of the pixel of interest is given a number indicating its position, and the position numbers are arranged in a predetermined order in the column and row directions, and the image is divided and labeled. Focusing on one pixel at the boundary of a region, by checking the label values of pixels in the vicinity of this pixel, the boundary of the region near this pixel is locally searched, and the method connects to the searched pixel and is located at the boundary of the region. The position number of the next pixel to be searched is determined based on the searched pixel, and the position number of the next pixel to be searched is determined according to the position number column of the pixel reference table specified by the position number. By searching for a pixel, the boundary of the area near the next pixel to be searched is locally searched, and the pixels located in the vicinity of each pixel at the boundary of the area are sequentially specified by the pixel reference table. Search locally according to the position number sequence, find the perimeter of the region and the common boundary length with the region adjacent to the region, and calculate the degree of connectivity between the adjacent regions based on the ratio of the perimeter and the common boundary length. A method for merging a plurality of regions, characterized in that the degree of connectivity is determined in memory and the degree of connection between the regions is determined. 23. Scan a window surrounding a predetermined number of pixels in the horizontal direction of the image-divided target area, count the number of pixels with a predetermined pixel value existing within the window for each scanning position, and calculate the number of pixels having the predetermined pixel value. When there are consecutive window positions where the number of pixels is less than or equal to a predetermined number, the window position where the number of pixels starts to become less than or equal to the predetermined number is determined to be the boundary between the target area and the background area, and the boundary of the target area is determined. A method for identifying boundaries of a target area, characterized by identifying boundaries. 24. Calculate the angle formed by each adjacent point of the point sequence that forms the boundary of the obtained target area, use the variance value of this angle as a standard for the smoothness of the point sequence, and calculate the point sequence of the boundary based on the angle. 24. The method for identifying boundaries of a target area according to claim 23, wherein the boundary of the target area is evaluated globally. 25. The points in the leftmost or rightmost point sequence of the boundary of the target area obtained from the image input last time, the points in the leftmost point sequence and the points in the rightmost point sequence of the boundary of the target area calculated from the image input this time. The Mahalanobis general distance between the points is determined to be the same as the point sequence in the current image that is located at a close distance to the point sequence in the previous image, and the point sequence at the left and right ends of the boundary of the target area is determined. 25. The method for identifying boundaries of a target area according to claim 23 or 24, wherein the method prevents reverse recognition of the target area.