JP2001033238A - Object recognizing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make recognizable an object with high reliability by extracting a brightness value of a small area included in a shielding area of a picked-up image shielded by a shielding means, judging the effectiveness of the image obtained by an image pickup means according to the brightness value. SOLUTION: A shielding part 40 can be attached to a first image pickup device 3, or may be formed independently of the image pickup device 3. The shielding part 40 shields a part of a lens included in the first image pickup device 3 or a part of a photo detecting element of an image sensor. Accordingly, an image is picked up with a part of the image picked up by the image pickup device 3 shielded. When an image obtained from the image pickup device 3 and converted to digital data is stored in an image memory 5, a shielding brightness extract part 41 extracts the brightness value of a pixel included in the shielded area from the image memory 5. An image determining part 42 determines whether the picked-up image is effective or not according to the extracted brightness value.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an optical object recognizing apparatus for recognizing an object outside a vehicle by using an image pickup means of a camera mounted on the vehicle.

[0002]

2. Description of the Related Art Japanese Unexamined Patent Application Publication No. 8-154205 discloses a technique for detecting the state of an image. An imaging device is described. This imaging apparatus is provided with a light-shielding unit that shields one line of the light receiving unit.
Smear is removed by subtracting the CCD output signal of the line.

[0003]

However, in addition to the above-described smear, the captured image may differ from the normal image due to electric noise, displacement of the lens / light receiving element, and the like. is there. If such an image is used in the object recognition processing, an object is erroneously recognized, a position of a white line is incorrectly recognized, or an inter-vehicle distance with another vehicle is incorrectly calculated. May affect control.

The present invention is intended to solve these problems, and it is an object of the present invention to determine the validity of an image obtained from an image pickup apparatus and perform more reliable object recognition. Aim.

[0005]

According to a first aspect of the present invention, there is provided an object recognizing apparatus having an object recognizing means for recognizing an object outside a vehicle. A light-shielding unit that shields a part of the light-receiving unit, and a brightness-value extracting unit that extracts a brightness value of a small region included in a light-shielded region shielded by the light-shielding unit in an image captured by the imaging unit. And an image determining unit that determines the validity of the image obtained by the imaging unit based on the luminance value of the extracted small region.

According to the first aspect of the present invention, it is determined whether the captured image is valid and the object recognition processing is performed based on the result, so that the accuracy of the object recognition can be improved.

According to a second aspect of the present invention, in the object recognition apparatus of the first aspect, the image determining means determines the validity of the image by using a plurality of images continuously captured by the image capturing means. Take the configuration.

According to the second aspect of the present invention, since the validity of the image is determined after continuously verifying the image a plurality of times, accurate and stable object recognition can be performed.

According to a third aspect of the present invention, in the object recognition apparatus according to the first or second aspect, the image judging means determines that the luminance value of the small area extracted by the luminance value extracting means is a predetermined value or more. The validity of the image is determined based on the number.

According to the third aspect of the present invention, the validity of an image is determined based on the number of small areas detected as errors due to noise or displacement, so that more accurate and stable object recognition can be performed. .

[0011] The invention of claim 4 is the invention of claims 1 to 3.
In the object recognition device according to any one of the above, the light receiving unit has a lens and a light receiving element, the light blocking means is arranged between the lens and the light receiving element or on the object side of the lens, the imaging means A configuration is adopted in which a plurality of lines of the obtained image are captured while being shielded from light.

According to the fourth aspect of the present invention, not only smear but also electric noise and displacement of the lens / light receiving element of the imaging device can be easily detected, and the validity of the image can be more accurately determined. be able to.

[0013]

Embodiments of the present invention will now be described with reference to the drawings. FIG. 1 is an overall block diagram of an object recognition apparatus according to one embodiment of the present invention. FIG. 2 is a view for explaining the principle of measuring the distance by the triangular measurement method used in this embodiment. First, a method of measuring a distance using a pair of imaging devices will be described with reference to FIG.

The image sensor 21 and the lens 23 constituting one of the pair of image pickup devices are spaced apart from the image sensor 22 and the lens 24 constituting the other image pickup device by a predetermined distance, that is, a base line length B in the horizontal or vertical direction. It is arranged in. Image sensors 21 and 22
Is typically a two-dimensional CCD, and may be a two-dimensional array of photosensors. In FIG. 2, for simplicity, only one of a plurality of lines of the two-dimensional image sensor is shown and described.

Considering nighttime use, it is preferable to use an imaging device using infrared rays. In this case, an infrared transmissive filter is placed in front of the lenses 23 and 24, and the object 20 is illuminated at a constant period using an infrared light source, and the image sensors 21 and 22 detect infrared rays reflected from the object 20. It is better to do it.

The image sensors 21 and 22 are arranged at the focal length f of the lenses 23 and 24, respectively. The image of the object at a distance a from the plane with the lenses 23 and 24 is
The image sensor 21 is formed at a position shifted by X1 from the optical axis of the lens 23.
If it is formed at a position shifted by X2 from the optical axis of
The distance a from the surfaces of the lenses 23 and 24 to the object 20 is
According to the principle of the triangulation measurement method, a = B · f / (X1 + X2)
Is required.

In this embodiment, since the image is digitized, the distance (X1 + X2) is digitally calculated. While shifting one or both of the images obtained by the image sensors 21 and 22, the sum of the absolute values of the digital values indicating the luminances of the corresponding pixels of the two images is obtained, and this is used as the correlation value. The shift amount of the image when the correlation value becomes the minimum value indicates the displacement between the two images, that is, (X1 + X2). Ideally, as shown in FIG. 2, the distance over which the two images must be moved relatively to overlap the two images obtained from the image sensors 21 and 22 is (X1 + X2).

As described below, in one embodiment of the present invention, a two-dimensional CCD or a two-dimensional CCD is used as an image sensor.
Use a three-dimensional photosensor array. In this case, 2
When the two-dimensional images obtained from the two image sensors are relatively shifted and the same correlation calculation as described above is performed, and the shift amount when the correlation value becomes minimum is obtained, the shift amount becomes (X1 + X2). Equivalent to.

Next, an overall configuration of an embodiment of the present invention will be described with reference to FIG. The first imaging device 3 corresponds to one of the imaging devices including the lens 23 and the image sensor 21 in FIG. 2, and the second imaging device 3 ′ corresponds to the other imaging device including the lens 24 and the image sensor 22 in FIG. Corresponds to the device. In this embodiment, the imaging region is divided into a plurality of windows W ₁₁ , W ₁₂ ,..., And the distance is measured for each window. Therefore, a two-dimensional image of the entire object is required. For this reason, the image sensors constituting the first and second imaging devices 3 and 3 'are realized by a two-dimensional CCD array or a two-dimensional photo sensor array.

Referring now to FIG. 6, FIG.
Travels in front of the own vehicle by the imaging device 3 or 3 ′.
FIG. 6B shows an example of an image obtained by capturing another vehicle.
The image of FIG. 6A is conceptually divided into a plurality of windows.
This is shown. FIG. 6 (b) shows rows and horizontal lines
10 rows x 15 columns for simplicity
C is shown separately. Each window is numbered
For example, W ₁₂Is the window in row 1 and column 2.
Show c.

An image of the object picked up by the image pickup devices 3 and 3 'is converted into an analog / digital converter (A / D converter) 4
Are converted to digital data by the image memory 5 and
And 5 'respectively. By the window cutout unit 13, an image portion corresponding to the window W ₁₁ is sent to the correlation calculation section 6 cut out respectively from the image memory 5 and 5 '. The correlation calculator 6 calculates the extracted 2
The above-described correlation calculation is performed by shifting the two images by a predetermined unit, and the shift amount when the correlation value is minimized is (X1 + X2). The correlation calculator 6 sends the value of (X1 + X2) thus obtained to the distance calculator 7.

The distance calculator 7 calculates a = B · f /
(X1 + X2) using the formula, obtaining the distance a ₁₁ to the object in the window W _11. The distance a ₁₁ thus obtained is stored in the distance storage unit 8. Similar calculation processing is sequentially performed for each window, and the distances a ₁₁ , a ₁₂ ,... Are stored in the distance storage unit 8.

The calculation of the distance can be executed only for windows necessary for the use of the road area determination unit 34 and the object detection unit 35 described below. Less than,
The distance to the target calculated for a certain window is called the measured distance of that window.

Since the resolution of the image data used in the above correlation calculation is determined by the pitch of the elements of the image sensor array, when a light receiving element having a relatively large pitch such as a photo sensor array is used, interpolation calculation between the pitches is performed. It is preferable to perform a process of increasing the density of the image data by performing the correlation calculation on the image data having the increased density.

In order to correct the characteristic change of the image sensor array due to the temperature, a temperature sensor may be arranged near the image sensor array, and the distance calculation may be corrected based on temperature information obtained from the temperature sensor. .

Next, with reference to FIGS. 1 to 4, light-shielding of the light-receiving portion of the imaging device for determining the validity of the captured image will be described. The light shielding unit 40 in FIG. 1 can be typically attached to the first imaging device 3, or may be formed independently of the imaging device 3. Light shield 40
Shields a part of the lens 23 included in the first imaging device 3 or a part of the light receiving element of the image sensor 21. Therefore, a part of the image captured by the first imaging device 3 is captured in a light-shielded state.

The light-shielding portion 40 can be realized by various methods, for example, by a light-shielding plate formed of a light-shielding metal material such as aluminum, copper, cobalt, and tungsten. Or you may implement | achieve by the light shielding film which formed the black mask on the transparent film.

Although the light-shielding portion 40 can directly form a light-shielding film on the light receiving element included in the image sensor, as shown by a line 25 in FIG. Placed between the image sensors 21 or as shown by line 26
It is preferable to dispose a part of the lens 3 on the object side of the lens 24 so as to shield light. By doing so, it is possible to detect the displacement of the lens or the light receiving element of the image sensor based on the captured image. Note that FIG.
The lines 25 and 26 shown in FIG. 2 conceptually show places where the light shielding unit 40 is arranged, and do not show the shape of the light shielding unit 40.

Further, as shown in FIG. 3, the light shielding unit 40 is provided with a lens so as to act as a "eave" for preventing an extra amount of light from entering the light receiving unit of the image pickup device due to the influence of pitching or rolling of the vehicle. Placed on the object side of
Preferably, it is formed so as to shield the upper side of the lens.

In this embodiment, the light-shielding section 40 is formed in the image pickup device 3 so that the upper 1 line to 60 lines of the picked-up image are shielded and picked up. FIG. 4A illustrates an image area captured by the first imaging device 3, and this region corresponds to a two-dimensional image sensor included in the imaging device 3. The two-dimensional image sensor
For example, since 512 light receiving elements are provided in the horizontal direction and 480 light receiving elements are provided in the vertical direction, the image area to be imaged is 512
It has 480 pixels. In this embodiment, the area from the first line to the 60th line of the image area is defined as a light-shielded area, that is, a light-shielded area 51, and the 61st to 480th lines are defined as the object recognition area to be subjected to the object recognition processing. 52.

The light-shielded area is one of
It is defined as a place that is not used for object recognition processing, that is, is not a region where an object on the road ahead of the vehicle is imaged. Further, in order to prevent an extra amount of light from entering the light receiving element as the "eave" as much as possible, it is preferable to define the light shielding area 51 above the image area as shown in FIG. Further, the plurality of lines of the image sensor are shielded from light so that electrical noise that is difficult to detect by shielding only one line can be easily detected.

FIG. 4B shows an example of an image of the front of the vehicle taken by the image pickup device 3 via the light shielding portion 40 thus formed. Light shielding area 5 of 1 line to 60 lines
Reference numeral 1 denotes a portion which is imaged while being shielded from light by the light-shielding unit 40. Since the luminance value of the pixel indicates zero or a value close to zero, the image is darkly imaged as shown in FIG. Also,
In the object recognition area 52, another vehicle running ahead is imaged.

FIG. 4 (c) shows an image area picked up due to an increase in the amount of light partially received, a poor contact, a component failure, an input from a power supply or a signal harness, or an electric influence due to radio waves or the like. Shows an example of an image containing noise.
Here, the noise is represented by a white square. The occurrence of noise increases the luminance value of some pixels. Therefore,
When the noise affects the light-shielding region 51, the luminance value of some of the pixels included in the light-shielding region 51 increases, and as shown in the regions 53 and 54 in FIG. Appears in region 51.

FIG. 4D shows an example of an image picked up in a situation where the lens is misaligned or the light receiving element constituting the image sensor is misaligned due to an impact or a change in temperature. As is apparent from the figure, since the position of the lens or the light receiving element is shifted with respect to the light-shielding region 51, the light-shielded region 51 and the dark portion region where the luminance value is zero or close to zero are imaged with a shift. I have. Therefore,
In the light-shielding area 51, an area 55 (a shaded portion)
Includes pixels whose luminance values have increased.

As described above, a pixel whose luminance value has increased in a light-shielded area due to a partially increased light amount or electric influence, and a position shift of a lens or a light receiving element is hereinafter referred to as an “error pixel”. I will call it.

Next, a method for judging the validity of an image based on the luminance values of the pixels in the light-shielded area 51 of the image captured via the light-shielding section 40 will be described. In this example,
The luminance value is represented as digital data having 256 gradations (between the value “0” representing pure black and the value “255” representing pure white).

In this embodiment, although the validity of an image is determined based on the luminance value of each pixel, the light-shielded area is divided into a plurality of small areas, and the image is determined based on the luminance value of each small area. The effectiveness may be determined. In this case, a plurality of pixels can be combined to form one small area, and the average luminance value of the pixels included in the small area can be used as the luminance value of the small area. Furthermore, this small area can be made the same size as the window used for performing the distance measurement described above.

Returning to FIG. 1, when an image obtained from the image pickup device 3 and converted into digital data is stored in the image memory 5, the light-shielded luminance extracting section 41 reads out the pixels included in the light-shielded area 51 from the image memory 5. Is extracted. In this embodiment, as shown in FIG. 4A, luminance values of 30,720 pixels corresponding to 1 to 60 lines are extracted.

The image determining unit 42 determines whether the captured image is valid based on the luminance value extracted by the light-shielding luminance extracting unit 41. As described above, when the pixel whose luminance value has increased is included in the light-shielding region 51, the pixel of the image captured as shown in FIG. Images can be smeared or electrical noise can affect the image,
Alternatively, the positions of the lens and the light receiving element may be shifted as shown in FIG. In such a case, if the captured image is used for the object recognition processing, the object may be erroneously recognized. Therefore, it is possible to determine the validity of the image and perform more reliable object recognition. To

First, the image judging section 42 detects an error pixel due to noise or misalignment by the error pixel detecting section 43 based on the luminance value of the pixel included in the light shielding area 51. The error pixel detection unit 43 compares the luminance value of a pixel included in the light shielding area 51 with a predetermined value, and if the luminance value of the pixel is larger than the predetermined value, detects the pixel as an error. In the example of FIG. 4C, pixels included in the noise regions 53 and 54 are detected as errors, and in the example of FIG. 4D, the pixels included in the displacement region 55 are detected as errors. Next, the error pixel detection unit 43 calculates the total number of pixels detected as errors in the light-shielded area 51.

In order to perform the processing at high speed, it is preferable to perform the processing of extracting the luminance value of the pixel and comparing it with a predetermined value in parallel. For example, a plurality of lines of pixels can be extracted and the pixels of each line can be processed in parallel.

If the number of pixels detected as an error by the error pixel detecting unit 43 is larger than a predetermined value, the invalid image candidate extracting unit 44 in the image determining unit 42 extracts a captured image as an invalid image candidate. If the number of error pixels is larger than a predetermined value, it indicates a situation in which a large number of noises have occurred in the captured image or a large positional shift of the lens or the light receiving element. Therefore, the captured image is set as an invalid image candidate. If the number of error pixels is smaller than a predetermined value, the error pixel can be considered to be due to an error included in the luminance value of the pixel, and thus the captured image is determined to be a valid image. As described above, by extracting an invalid image candidate based on the number of pixels detected as an error, it is possible to extract an image that is likely to cause erroneous recognition in object recognition.

In another embodiment, the invalid image candidate extracting section 44
May sum the luminance values of the pixels detected as errors, and extract the captured image as an invalid image candidate if the total luminance value is larger than a predetermined value. Alternatively, the invalid image candidate extracting unit 44 calculates the average value of the luminance values of all the pixels in the light-shielded area, and when the average value is greater than zero or a value near zero, the image captured this time is regarded as an invalid image candidate. You may make it extract. Furthermore, instead of the average value, the sum of all the brightness values of the pixels in the light-shielded area may be calculated, and when the sum is larger than a predetermined value, the image may be regarded as an invalid image candidate.

Next, when the invalid image candidates are successively extracted a predetermined number of times or more, the final judgment unit 45 in the image judgment unit 42 finally judges that the image captured this time is invalid. In this embodiment, an image is captured by the imaging device 3 at a cycle of, for example, 100 milliseconds, and the validity of the image is determined. Therefore, for example, when an invalid image candidate is continuously extracted for ten cycles, the image captured this time is finally determined to be invalid. In this way, in the case of a permanent failure that is not automatically repaired, such as a displacement of a lens or a light receiving element, the object recognition process can be stopped to prevent erroneous recognition of an object. Also,
In the case of a momentary error such as an instantaneous increase in the amount of light or noise appearing due to the effect of a momentary electromagnetic wave, the image is not determined to be an invalid image. Can be eliminated.

If the final determination section 45 determines that the image is invalid, it outputs a fail signal indicating that the image captured this time is invalid. The output fail signal can be sent to, for example, the object recognition unit 14 in FIG. The object recognizing unit 14 receives the fail signal, and may not perform the object recognition process on the image determined to be invalid, or may cancel the already started object recognition process.

Further, the final determination section 45 can send a fail signal to the vehicle control section 18. Vehicle control unit 16
Can receive a fail signal and not enter a vehicle control mode based on the result of object recognition. Further, by outputting the fail signal by the final determination unit 45, the system can be switched to perform the object recognition processing using only the millimeter wave by the laser radar.

In this embodiment, when it is not determined that the image is invalid, the final determination unit 45 outputs a pass signal indicating that the image captured this time is determined to be valid. In this case, for example, the object recognition unit 14 can start the object recognition processing after receiving the pass signal, and the vehicle control unit 18 enters the vehicle control mode after receiving the pass signal. You can do so. However, the final determination unit 45 may not output any signal when determining that the image is valid.

In order to perform the object recognition processing at a higher speed, it is preferable to execute the above-described measurement of the distance to the object and the processing of judging the validity of the image in parallel.

In another embodiment, the invalid image extracting unit 44
When the position of the error pixel in the image as well as the luminance value of the error pixel in the light-shielded region of the image extracted as the invalid image candidate is simultaneously extracted, and the final determination unit 45 extracts the error pixel at the same position a plurality of times, It may be determined that the image is invalid. In such a case, there is a possibility that the failure is a permanent failure such as a displacement of the lens or the light receiving element, so the final determination unit 45 outputs the position of the error pixel in the image simultaneously with the output of the fail signal. hand,
The fault location can be displayed on some monitor.

In this embodiment, the validity of an image is determined based on an image taken by one image pickup device 3. However, both the pair of image pickup devices 3 and 3 'used for measuring the distance are shielded from light. And the validity of the image may be determined using the two captured images. In this case, the validity of the images can be determined in parallel for each of the two captured images, and the validity of the final image can be determined based on the result, or the luminance value of the two images can be determined. May be added and combined, for example, and the validity of the image may be determined based on the luminance value of the combined result.

FIG. 5 is a flowchart executed by the light-shielding luminance extracting section 41 and the image determining section 42 to determine whether or not the captured image is valid based on the luminance values of the pixels included in the light-shielding area 51. In the initialization of step 501, the number of error pixels is cleared to zero. The number of error pixels is the number of pixels included in the light-shielded area and detected by the error pixel detection unit 43 as an error. Since the number of error pixels is calculated for each captured image, step 5
Initialize with 01.

At step 502, the luminance value of the pixel included in the light-shielded area 51 is extracted. The extracted luminance value is
In step 503, a predetermined value P1 (for example, P1 = 100
If it is larger than the predetermined value P1, the process proceeds to step 504, and 1 is added to the number of error pixels. In this way, the brightness value comparison process is repeated for the pixels included in the light-shielded area, and all the pixels included in the light-shielded area and having a brightness value larger than a predetermined value are detected as errors. If the luminance value comparison processing has been performed for all the pixels included in the light-shielded area in step 505, the process proceeds to step 506.

In step 506, the sum of the number of pixels detected as an error in the light-shielded area of the currently captured image counted in step 504 is larger than a predetermined value P2 (for example, P2 = 20). For example, the image captured this time is determined to be an invalid image candidate, and 1 is added to the number of invalid image candidates in step 507. Predetermined value P2
If it is below, it is determined that the image captured this time is valid, and the process proceeds to step 508 to clear the number of invalid image candidates to zero. The number of invalid image candidates indicates the number of images successively determined to be invalid image candidates for a plurality of captured images, and is therefore cleared to zero when the images are determined to be valid images. If it is determined that the image is valid, step 5
Proceeding to step S11, a path signal indicating that the image captured this time is determined to be valid is output.

In step 509, the number of invalid image candidates is a predetermined number P3 (for example, P3 can be set to 10).
If this is the case, it indicates that the invalid image candidate has been successively extracted three times, so that the image captured this time is finally determined to be an invalid image, and the process proceeds to step 510. If the number of invalid image candidates does not reach the predetermined number P3, the process exits this routine. In step 510, a fail signal indicating that the image captured this time is determined to be invalid is output.

Next, an object recognition method will be described. The object recognition unit 14 of FIG. 1 is provided from the distance of each window stored in the distance storage unit 8 and the image memory 5,
The object is recognized based on the image data corresponding to the object recognition area 52 shown in FIG. FIG. 7 is a block diagram illustrating a configuration of the object recognition unit 14. The object recognizing unit 14 in this embodiment uses a method of determining a road area from an image and determining an object that is not a road area as an object.

The determination of the road area in the image will be described with reference to FIGS. As described above, FIG. 6B shows the object recognition area 5 shown in FIG.
The road area is determined for the object recognition area 52 in response to No. 2. In FIG. 6B, although the image area is divided into windows of 10 rows × 15 columns, the image area is actually very finely divided. In order to determine the road area more precisely, each window can be composed of one pixel. Or, when a plurality of pixels are combined,
One window may be configured. Each window may be the same size as the windows used to measure the distances described above, or may be different sizes.

When an image obtained from the imaging device 3 and converted into digital data is stored in the image memory 5, the window cutout unit 13 shown in FIG. Cut out the window. The luminance extracting unit 31 acquires a plurality of luminance values from the clipped window.

The reason why the luminance value of the window included in the image region immediately before the vehicle is obtained is that the image region immediately before the own vehicle is very likely to be a road. This is because even if there is a sign area such as a character / white line on the road surface, the original luminance value of the road can be obtained. Which window of the input image is acquired as a plurality of windows included in the image area immediately before the host vehicle is determined in advance according to the size of the vehicle, the position of the imaging device in the vehicle, and the like.

Next, in order to extract the original road luminance value, the luminance value of the window including the sign on the road surface is removed from the plurality of acquired luminance values. For example, if the bottom row of windows in the image contains several windows for signs on the road, the brightness of the signs on the road is generally very different from the brightness of the road, so the brightness value of Variations occur. Therefore, it is possible to average the luminance values of the windows in this row, and remove luminance values having values separated from the average value by a predetermined value or more.

Alternatively, since the color of the sign on the road surface is mainly white or yellow, which is very different from the color of the road, the luminance value in the range corresponding to white or yellow may be removed. Furthermore, based on the reference luminance value extracted from the previously input image, it may be estimated whether the luminance value acquired from the currently input image is the luminance value of the original road.

After removing the luminance value of the window including the sign on the road surface, the luminance extracting unit 31 extracts a reference luminance value based on the remaining luminance value and stores the reference luminance value in the luminance storage unit 32.
For example, one or more of the remaining luminance values can be selected and stored as a reference luminance value. Alternatively, a value obtained by averaging a plurality of luminance values can be stored as one reference luminance value. As described above, the luminance value is represented as digital data having 256 gradations (between pure black “0” and pure white “255”).

Next, the window extracting section 13 extracts another window from the image, and the luminance extracting section 31 extracts the luminance value of the window. The luminance comparing unit 33 compares the extracted luminance value with the reference luminance value stored in the luminance storage unit 32.

When each window is composed of a plurality of pixels, the average of the sum of the luminance values of the respective pixels can be obtained, and the average value can be extracted as the luminance value of the window.

Here, the process of extracting and comparing the luminance values is preferably executed in parallel with the above-described process of calculating the distance, and in parallel with these processes, the effectiveness of the above-mentioned image is checked. It is preferable to make a judgment.

The road area determining section 34 determines a road area based on the comparison result received from the luminance comparing section 33. As described above, in this embodiment, the object recognition unit 14
Starts the process after receiving a pass signal indicating that the captured image is valid from the image determination unit 42. If the received comparison result is within the predetermined range, the road area determination unit 34 determines that the window is a road area. This is because the brightness of the road area is similar to that of a vehicle traveling ahead and the like. The luminance value of the window determined to be the road area is stored in the luminance storage unit 32 as a new luminance value.

Referring to FIG. 6B, an example of determination of a road area based on a luminance value will be described. The windows W _A7 and W _A9 (windows shaded with oblique lines) including the image area immediately before the vehicle are cut out by the window cutout unit 13, and the brightness extraction unit 31 extracts the brightness values L1 and L2 of each window, and The luminance value is stored in the luminance storage unit 32. Next, the window W _A6 adjacent to the window W _A7 is cut out, and the luminance extracting unit 31 extracts the luminance value of the window W _A6 . The luminance comparing unit 33 compares the extracted luminance value with the reference luminance value L1. If the comparison result is within a predetermined range (for example, a range of ± 3 with respect to the reference luminance value can be set as the predetermined range), the road area determination unit 34 determines that the window _WA6 is a road area, and the luminance values of W _A6, stores the luminance memory unit 32 as a new reference luminance value L3.

Next, the window W _A5 adjacent to the window W _A6 is cut out, and the brightness value of the window W _A5 is extracted by the brightness extraction unit 31. The luminance comparing unit 33 compares the extracted luminance value with the reference luminance value L3. Road area determination unit 34, if the result of comparison is within a predetermined range, the window W _A5 is determined that the road area, the luminance value of the window W _A5, and stores the luminance memory unit 32 as a new reference luminance value L4. In this way, windows are sequentially cut out from the image, and the road area is determined by comparing the luminance values for each window.

The window cut out by the window cutout unit 13 is preferably located near a window having a reference luminance value. Specifically, if the reference luminance value is the luminance value of the window W _A6, preferably to compare the luminance value by cutting a window belonging to the window or adjacent rows belonging to the same row as the window W _A6. This is because if the difference between the measured distances of the two windows to be compared is large, the luminance value may be substantially different even on the same road.
According to this embodiment, even when the brightness of the road in the image changes according to the distance from the host vehicle, the road area can be accurately detected.

As in the above-described embodiment, the luminance value first extracted from the window included in the image area immediately before the vehicle (in the above example, the luminance value determined as the road area is not used as a new luminance value). , L1 and L2) as a fixed reference luminance value, and comparing the luminance value of each window of the image with the reference luminance value, the road area can be determined.

Further, in the above embodiment, the luminance value is extracted based on one image obtained from one image pickup device 3, but the luminance value is extracted from two or more image pickup devices necessary for the distance measurement described above. The extraction may be performed using two or more obtained images. For example, the reference luminance value L1 can be extracted from the image obtained by the first imaging device 3, and the reference luminance value L2 can be extracted from the image obtained by the second imaging device 3 '.

The processing for judging the road area by comparing the luminance values is preferably performed in some parallel processing. For example,
The luminance values of the windows W _{A1 to} W _A6 and W _{91 to} W ₉₇ are collectively compared with the reference luminance value L1, and then the windows W _{81 to} W _81
The window can be processed on a row-by-row basis, such as by comparing the luminance value of _{87 with} the luminance value of window W ₉₃ that has become the new reference luminance value. Alternatively, for high-speed processing, processing the left half window of the image using the reference luminance value L1 as a base point, processing the right half window of the image using the reference luminance value L2 as a base point, and processing both in parallel. Good.

Further, an area surrounded by the image area determined to be a road area can be automatically determined to be a road area. Thus, for example, even if the area surrounded by the area determined to be the road area is a sign area having a luminance different from that of the road, the sign area can be determined as the road area. The size of the area surrounded by the road area and the size of the area that can be determined as the road area depend on the size of the object to be detected.

As described above, since the road surface itself is detected based on the brightness value, the road area can be determined even when the vehicle is inclined by pitching or rolling, or is running on a slope or a bank. Can determine that there is no other vehicle or obstacle.

Here, the sign area on the road can be accurately recognized using the measured distance of the window.
The road area determination unit 34 acquires from the distance storage unit 8 the measured distance of the window for which the comparison result is determined not to be within the predetermined range, and determines whether this distance is a distance to a road.
If the distance to the road, this window can be determined as a sign area on the road.

The distance of a window to a road can be estimated from the measured distance of another window determined as a road area (ie, the measured distance to a road). For example, the distance to the road can be estimated to be the same for all windows included in the row to which the other window belongs. Further, the distance to the road can be estimated for each row of the window from the measured distance of the window determined to be a road. Therefore, the road area determination unit 34
Can determine whether the image area of the window is a sign area on the road by comparing the distance actually measured for the window with the estimated distance to the road.

[0076] For example, as shown in (b) of FIG. 6, the window W ₉₅ includes a character on the road surface. Road area determination unit 3
4 receives the comparison results for the window W _95,
The comparison result is not within a predetermined range, acquires the measured distance of the window W ₉₅ from the distance memory unit 8. Further, for example, the measured distance of another window W ₉₃ belonging to the same row as the window W ₉₅ and determined to be a road area is acquired from the distance storage unit 8. Result of comparing the two distances, the substantially same distance, determines an image area of the window W ₉₅ and indicator area on the road surface. By repeating such determination, FIG.
The sign “60” on the road as shown in FIG.

As described above, since the sign area on the road can be recognized using the measured distance, it is also possible to control the vehicle so as to draw the driver's attention to, for example, excessive speed or changing lanes. it can.

The determination of the road area described above may be performed for the entire area of the object recognition area 52, or may be performed for only a part of the area. For example, with respect to the previously input image, the process can be executed only for an image region newly input as an image along with the travel of the host vehicle. Furthermore, the road area may be determined by using a preset road model of the car navigation system. In this way, by limiting the image area to be determined, it is possible to more efficiently determine the road area.

Since the road area has been determined, the windows in the image are classified into those in the road area and those in the area other than the road area. The road area determination unit 34 can output a road area composed of a window determined as a road area in the form of an image as needed. FIG. 6C shows an example of this output image, in which the detected road area is displayed in a solid color.

The object detecting section 35 detects an object on the road based on the road area determined by the road area determining section 34. Since the road area has been determined, an object can be detected by extracting a window located in front of the road area and not determined as the road area.

For example, since the road area is determined as shown in FIG. 6C, the road area is traced forward, and the windows W ₅₇ , W ₅₈ , W ₅₉ not determined as the road area are extracted. As shown in FIG. 6B, these windows include another vehicle traveling ahead as an object. The object detection unit 35 acquires the measured distances of these windows from the distance storage unit 8. From the acquired measurement distance,
The distance from the own vehicle to another vehicle can be detected.
Furthermore, the position of the window W _57, W _58, W ₅₉ of the object area of the window W ₆₆ to _W-6A it is determined that the road area, the other vehicle may be determined to be in the middle of the road. Thus, the object detection unit 35 can detect the inter-vehicle distance to another vehicle traveling ahead based on the detected distance to the object.

Further, the object detecting section 35 can calculate the relative speed with respect to another vehicle by using a formula of (current distance-previous distance) / detection time interval. The detection time interval is
The time difference between the previous measurement and the current measurement, which is 100 milliseconds in this embodiment. In this way, the object detection unit 35 stores information such as the following distance, the position on the road, and the relative speed of the detected object in the object storage unit 15.

The vehicle control unit 16 performs the preceding vehicle tracking control of the own vehicle based on the information stored in the object storage unit 15 and the information from the own vehicle speed detecting device 18 and the yaw rate detecting device 17. Control such as output, abnormal approach alarm, and forced deceleration control. For example,
When a driver can be warned based on the inter-vehicle distance to another vehicle ahead calculated by the object detection unit 35 and stored in the object storage unit 15, and when there is an obstacle in front of the road that hinders the traveling of the vehicle. In this case, an alarm can be sounded to alert the driver.

In another embodiment, apart from the object recognition method described above, in another embodiment, for example, the object recognition process is periodically executed in 100 milliseconds, so that the position of the previously recognized object and the relative position to the object are determined. Estimating the position of the current object based on the speed, extracting a window that is within a predetermined range with respect to the estimated object position and that overlaps with the estimated object, and recognizes the object in the image captured this time You can also.

Further, in another embodiment, the validity of an image can be determined together with the result of the object recognition processing. For example, the image determination unit 42 determines the image based on the luminance value of the pixel in the light-shielding region 51, and sends a fail signal to the vehicle control unit 18 when it determines that the image is invalid. On the other hand, the object recognition unit 14 performs the object recognition process a plurality of times at a cycle of, for example, 100 milliseconds, and outputs a fail signal when the variation in the distance to the recognized object is large or when the variation in the position of the white line is large. Output to vehicle control 18. When the vehicle control unit 18 receives one of the fail signals,
Alternatively, when both fail signals are received, the vehicle control based on the result of object recognition using the image captured this time is stopped. Alternatively, what the fail signal is based on can be displayed on some monitor, for example.

Further, the object recognizing section 14 may detect a pixel error in the object recognizing area 52. For example, the object recognizing unit 14 includes the luminance extracting unit 31
A window (or pixel) whose luminance value is zero or near zero is extracted. If a window with a luminance value of zero or near zero is extracted at the same location a plurality of times, the object recognizing unit 14 outputs a fail signal. This is because it can be determined that some abnormality has occurred in the portion of the lens or the light receiving element corresponding to the region where the window at or near zero is extracted, and the corresponding region is shielded from light and imaged. The vehicle control unit 18 can receive the output fail signal, stop the vehicle control based on the object recognition, or display some information on a monitor. Thus, by judging the validity of the image in combination with the object recognition processing, the accuracy of the object recognition processing can be further improved.

The processing of image validity determination and object recognition is executed in the same cycle, for example, every 100 milliseconds. As described above, for example, when the image is extracted as an invalid image candidate ten times in a row, the image determining unit 42 determines that the image is invalid and outputs a fail signal. In this case, it is highly likely that not only the image captured this time but also the nine images captured nine times before that are invalid, and the determination of an invalid image is highly reliable. Therefore, the vehicle control unit 1
It is preferable that the control unit 8 actually controls the vehicle after receiving the output of the fail signal from the image determination unit 42, not based only on the recognition result by the object recognition unit 14.

The correlation calculation unit 6, distance calculation unit 7, distance storage unit 8, window cutout unit 13, light-shielding luminance extraction unit 41, image determination unit 42, object recognition unit 14, object storage unit 15, and vehicle shown in FIG. The control unit 16 includes a central processing unit (CPU), a read-only memory that stores a control program and control data, and a random access memory (RAM) that provides an operation work area of the CPU and can temporarily store various data. Can be configured. The distance storage unit 8 and the object storage unit 15 can be realized using different storage areas of one RAM. Further, a temporary storage area of data required for various calculations can be realized by using a part of the same RAM.

Further, the object judging device of the present invention can be connected to an electronic control unit (ECU) of the engine, a brake control ECU and other ECUs via a LAN, and the output from the object judging device can be used for the overall control of the vehicle. it can.

[0090]

According to the first aspect of the present invention, it is determined whether or not the captured image is valid, and the object recognition process can be performed based on the result, so that the accuracy of the object recognition can be improved. .

According to the second aspect of the present invention, since the validity of an image is determined after continuously verifying the image a plurality of times, accurate and stable object recognition can be performed.

According to the third aspect of the present invention, since the validity of an image is determined based on the number of windows detected as an error due to noise or displacement, more accurate and stable object recognition can be performed. .

According to the fourth aspect of the present invention, it is possible to easily detect not only smear but also electric noise and a displacement of a lens / light-receiving element of an image pickup device, and more accurately determine the validity of an image. It can be carried out.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall configuration of an embodiment of the present invention.

FIG. 2 is a view for explaining a principle of measuring a distance by a triangular measurement method and an arrangement of a light shielding unit.

FIG. 3 is a diagram in which a light-shielding portion is arranged on an imaging object side of a lens.

FIGS. 4A and 4B show a case where the light-shielding means according to the present invention is used, and FIG. The figure which shows the example of the image of FIG.

FIG. 5 is a flowchart for determining the validity of a captured image.

FIG. 6 shows (a) a captured image,
FIG. 3B is a diagram illustrating an image divided into windows for determining a distance and a road area, and FIG.

FIG. 7 is a block diagram showing details of an object recognition unit 14 in FIG. 1;

[Explanation of symbols]

 3, 3 'imaging unit 40 light shielding unit 41 light shielding luminance extraction unit 42 image determination unit 14 object recognition unit 16 vehicle control unit 51 light shielding area 52 object recognition area

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Masakazu Saka 1-4-1 Chuo, Wako-shi, Saitama Prefecture Inside Honda R & D Co., Ltd. (72) Morishi Nishigaki 1-4-1 Chuo, Wako-shi, Saitama No. F-term in Honda R & D Co., Ltd. (reference) 2F065 AA02 AA03 AA14 BB15 CC11 DD03 EE01 FF09 FF24 HH13 JJ03 JJ26 LL00 LL21 MM02 QQ03 QQ24 QQ25 QQ27 QQ32 QQ36 QQ41 UU05 2F112 AC03 FA01 FA07 FA13 FA03 FA13 BA15 CC03 DA07 DB03 DC02 DC22 DC34 5H180 AA01 CC04 CC07 CC24 LL01 LL08

Claims (4)

[Claims] An object recognizing device having an object recognizing means for recognizing an object outside a vehicle, at least one image capturing means having a light receiving part, a light shielding means for shielding a part of the light receiving part, and the image capturing means A luminance value extracting unit that extracts a luminance value of a small region included in the light-shielded region shielded by the light-shielding unit in the captured image; And an image determining unit that determines the validity of the obtained image. 2. The object recognition apparatus according to claim 1, wherein said image judging means judges the validity of the image by using a plurality of images continuously picked up by said image pickup means. 3. The image judging means judges the validity of an image based on the number of small areas whose luminance values of the small areas extracted by the luminance value extracting means are equal to or greater than a predetermined value. Alternatively, the object recognition device according to claim 2. 4. A plurality of lines of an image obtained by said imaging means, wherein said light receiving section has a lens and a light receiving element, and said light shielding means is arranged between said lens and said light receiving element or on an object side of said lens. The object recognition device according to any one of claims 1 to 3, wherein the object is imaged while being shielded from light.