JP2000207563A - Image recognizing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a reliability in an image recognizing device. SOLUTION: The image of a forward travelling direction is picked-up by a camera 1 which is mounted on a vehicle such as an automobile, a white line to display the travelling lane of a road is recognized and processed, driving is supported and automatic driving is executed. A condition detecting means arithmetically processes image data from the camera 1 and detects a recognition processing impossible condition such as a state where the image is too bright by backlight or after the rain, a state where the image is too dark at night or the like, a state where the image is the low contrast one owing to twilight or snow or a state where sumearing occurs owing to backlight or the like. An announcing means 12 announces such a recognition processing impossible condition. Therefore, it can be judged whether or not an abnormality occurs because of a fault in the camera 1 or not and an input image to the camera is abnormal or inappropriate.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image recognizing apparatus for recognizing an object or the like based on image data picked up by a camera.
Based on image data from an in-vehicle camera mounted with the optical axis directed in front, side, or rear of the vehicle, for example, a white line representing a driving lane drawn on a road and the presence or absence of other vehicles are recognized and driving is performed. The present invention relates to driving assistance for notifying a driver or controlling driving of a vehicle, and an image recognition device suitably implemented in an automatic driving device.

[0002]

2. Description of the Related Art In such an image recognition device for a driving support and automatic driving device, when an object to be recognized, for example, a white line on a road or a preceding vehicle cannot be image-recognized by image data from a camera, an image is not recognized. You have determined that it is not recognizable.

FIG. 20 is a flowchart for explaining the operation of the above-mentioned prior art. The process proceeds from step g1 to step g2, where image data from the camera is input to processing means implemented by a computer or the like. In step g3, a recognition process of recognizing an object such as a white line indicating a driving lane drawn on the road is performed using the image data. When the recognition processing is possible in step g4, a signal indicating that the white line can be recognized is derived in step g5. In step g7, the operation for driving assistance, automatic driving, and the like is performed using the signal. Will be implemented. In step g6, a signal indicating that the recognition processing of the white line is impossible is derived, and thereby the operation of the driving support and the automatic driving is stopped.

In such prior art, the condition that the image recognition processing is not possible depends on the surrounding environment such as rain or snow, or the camera or the circuit for recognizing and processing the image data from the camera breaks down. It is impossible to judge whether it is abnormal. Therefore, the result is that the reliability of the image recognition device is poor.

[0005] Further, the recognition performance of the image recognition device is improved,
That is, when the application range is expanded, the recognition rate of the object is improved, but at the same time, false recognition may increase. Therefore, the operator needs to know whether the recognition processing of the image data from the camera can be appropriately performed or whether the recognition processing is impossible.

[0006] In particular, in the aforementioned driving support and automatic driving device,
It is necessary from the viewpoint of safe driving to accurately determine whether the recognition processing of the image data from the camera can be performed properly, otherwise, it is possible to use driving assistance and automatic driving devices Will be gone.

[0007]

SUMMARY OF THE INVENTION An object of the present invention is to perform a target object recognition process using image data obtained by a camera in applications such as driving assistance and automatic driving devices. An object of the present invention is to provide an image recognition device for determining whether or not the image recognition is performed.

[0008]

SUMMARY OF THE INVENTION The present invention provides a camera, condition detecting means for processing image data from the camera to detect a condition in which recognition processing is impossible, and responding to an output of the condition detecting means. Notification means for notifying the condition that the recognition processing is not possible, and recognition processing means for responding to the output of the condition detection means and performing recognition processing of image data from the camera when the recognition processing is possible. An image recognition device characterized by the following.

According to the present invention, the condition detecting means
It is determined whether or not the recognition processing of the image data from the camera can be performed by the recognition processing unit. If the recognition processing is not possible, the condition, that is, the cause is detected, and the notification unit notifies. The notification means may be a visual display means having a two-dimensional display surface such as a liquid crystal or a cathode ray tube, or may be an acoustic display means having a voice synthesis circuit and a speaker. It may be a means. So for example driving assistance,
In the image processing device for an automatic driving device, if the recognition processing unit cannot recognize the preceding vehicle imaged by the camera, it may not be able to recognize the absence of the preceding vehicle or the image data from the camera is abnormal. Alternatively, the operator can know whether the operation cannot be recognized due to improper operation. Thus, the reliability of the image recognition device is ensured.

Further, the present invention provides a camera, condition detecting means for calculating image data from the camera and detecting a condition in which recognition processing is impossible, and responding to an output of the condition detecting means to input an image to the camera. Correction means for correcting an image or image data from a camera, and recognition processing means for recognizing image data from a camera in which an input image has been corrected by the correction means or image data corrected by the correction means An image recognition apparatus characterized by including:

According to the present invention, the condition detecting means
When it is determined that the recognition processing unit cannot recognize the image data from the camera, the correction unit corrects the image input to the camera, and for example, (a) adjusts the aperture or shutter speed of the camera to an appropriate value. To correct the exposure condition, or (b) a camera is mounted in the cabin of the car,
In a configuration in which an image of the front of the vehicle is taken, a wiper for wiping the outer peripheral surface of the windshield is automatically operated to correct a normal front situation by the camera. The correcting means may be configured to correct the image data from the camera. For example, (c) when the image data from the camera is an analog signal representing shading, the image data is converted to a low contrast signal by the condition detecting means. When it is determined that the analog signal is converted to a digital signal, the sensitivity of the analog / digital conversion is increased, that is, the analog signal is converted into a digital signal for each small difference of the analog signal. Can be corrected.

By using the image data from the camera whose input image has been corrected in this way, or by correcting the image data from the camera, the recognition processing means can perform the recognition processing of the image data. When the condition detection unit determines that the image data cannot be recognized, the correction unit can expand the area of use as a corresponding environment and improve the performance of the image recognition device.

The present invention also provides a camera, a sensor for detecting an environment surrounding the camera, recognition processing means for performing recognition processing of image data from the camera, and a recognition processing means responsive to an output of the sensor. Condition detecting means for detecting a condition in which recognition processing of the image data is not possible, and notifying means for notifying a condition in which recognition processing is not possible in response to an output of the condition detecting means. An image recognition device.

According to the present invention, the surrounding environment of the camera is detected by the sensor, and whether the surrounding environment detected by the sensor is a condition that the recognition processing means cannot recognize the image data from the camera. Is determined by the condition detecting means, and when the recognition processing is not possible, there is no need to perform arithmetic processing on the image data from the camera in order to detect and determine the condition that the notifying means cannot perform the recognition processing. Since the output of the sensor is used, the amount of calculation processing can be reduced, and the condition for not being able to perform calculation processing can be quickly determined.

Further, according to the present invention, when the condition detected by the sensor output and the condition detected by performing arithmetic processing on the image data from the camera coincide with each other, an output representing the condition is provided. Is provided to the notification means.

According to the present invention, the condition that the recognition processing means cannot recognize the image data is detected not only by the output of the sensor but also by calculating the image data from the camera. When the condition of non-recognition process obtained in this way matches, the condition of non-recognition process is notified by the notification means, so that the determination of the condition of non-recognition process is achieved with high accuracy. Will be able to do it.

[0017]

FIG. 1 is a block diagram showing the overall configuration of a driving support and automatic driving device provided with an image recognition device according to an embodiment of the present invention. The image data from the camera 1 is provided to a processing circuit 2 implemented by a microcomputer, whereby the control circuit 3 controls the internal combustion engine 4 that drives the vehicle body of the automobile and controls the braking means 5. The camera 1 detects a preceding vehicle traveling ahead of the own vehicle in the traveling direction, and measures the distance between the preceding vehicle and the own vehicle by the processing unit 2.
The control circuit 3 controls the distance so that the distance is maintained at a predetermined value.
Controls the internal combustion engine 4 and the braking means 5, so that when the preceding vehicle travels, the own vehicle also travels, and when the preceding vehicle stops, the automatic driving is performed so that the own vehicle stops. Further, the processing means 2 recognizes the image data captured by the camera 1 and detects a white line representing a driving lane drawn on a road ahead of the vehicle. This detected white line is
When the vehicle fluctuates to the left and right in a short cycle, it is determined that the driver is dozing, and for example, sound is generated by sound display means such as a buzzer, etc.
Provide driving assistance.

FIG. 2 is a simplified side view of the vehicle body 6 of the automobile. Inside the cabin 8 where the driver's seat 7 is provided,
A reflecting mirror 9 is provided as a rearview mirror for viewing behind the vehicle body 6 in the running direction, and the camera 1 is provided in front of the reflecting mirror 9 in the running direction (to the left in FIG. 2). Further, a windshield 10 is disposed in front of the camera 1. The outer surface of the windshield 10 can be wiped by the wiper 11.

FIG. 3 is a flow chart for explaining the operation of the processing means 2. Step a1 to Step a
In step 2, image data of the scenery ahead of the vehicle body 6 in the traveling direction captured by the camera 1 is input to the processing circuit 2.
In step a3, image data from the camera 1 is subjected to arithmetic processing to detect and determine a condition under which recognition of an object that is a white line drawn on a road is not possible. When the recognition processing of the white line object is possible in step a4, the recognition processing of the object is performed in step a5. Step a6
In, an object recognition process is performed, and it is determined whether the object has been recognized. If a white line on the road is detected, in step a7, a signal indicating that the white line as the object is recognized is derived. Conversely, if an object that is a white line is not recognized, the vehicle body 6 may be running off the driving lane of the road, and at this time, a signal indicating that the white line has not been recognized is output by a step. Derived in a8. At step a9, when the white line is recognized, it can be determined that the vehicle body 6 is traveling on the traveling lane. At this time, the control circuit 3 controls the internal combustion engine 4 and the braking means 5 to automatically drive the vehicle. While performing the operation, the driving assist operation for preventing the falling asleep is performed. When the white line is not recognized in step a8, the automatic driving operation by the control circuit 3 is stopped.

When a condition that the object recognition process in step a5 is impossible is detected in step a4,
In step a9, a condition that the recognition processing is impossible is notified by the notifying unit 12. The notification means 12
For example, it may be a means for performing visual display on a two-dimensional display surface such as a liquid crystal or a cathode ray tube, or an acoustic display means having a voice synthesis circuit and a speaker. May be. The conditions under which the recognition process is not possible include, for example, the conditions shown in Table 1.

[0021]

[Table 1]

FIG. 4 is a flowchart showing a part of the operation of the processing means 2 shown in FIG. 1 in more detail. FIG.
Step b2 is the same as step a3 in FIG. 3, and is a step for determining whether or not recognition processing is impossible. The specific operation of step a4 in FIG. 3 is shown in steps b3 to b10 in FIG. The operations in steps b3 to b10 correspond to the phenomena 1 to 8 of the image shown in Table 2, respectively.

[0023]

[Table 2]

FIG. 5 is a front view showing an image 16 captured by the camera 1 and displayed on the two-dimensional display surface of the notifying means 12. On the entire screen 17 of the two-dimensional display surface 16,
A road portion 18 indicated by hatching and an own vehicle body portion 19 which is a front portion of the vehicle body 6 are included.

At step b2 in FIG. 4, the judgment operation shown in Table 2 is performed. In the next step b3, it is determined whether the image captured by the camera 1 is too bright based on unrecognizable conditions such as backlight or after rain. For this determination, the determination operations 1a to 1c in Table 2 are executed.

FIG. 6 is a diagram for explaining the judging operation 1a. When it is determined that the image of the entire screen is too bright due to backlight as shown in FIG. 6A, the density histogram shown in FIG. In this density histogram, the horizontal axis represents gradations 1 (black) to 256 (white), and the vertical axis represents the number of pixels having that density. Image data consisting of 8 bits is obtained for each pixel of the screen. FIG.
In the density histogram of (2), it can be seen that the tendency of the luminance value, that is, the density distribution, is almost white. In such a density histogram, for example, when the number of pixels having a density of 201 or more is a predetermined value or more, it can be determined that the image is too bright.

The luminance value of the pixels constituting the screen, that is, the gradation of the density, indicates the brightness when the image data is converted into a digital value, that is, the density, and in the case of a monochrome image, 8 pixels per pixel. The luminance value is represented by the value of the bit resolution 0 to 255. The density histogram is a graph in which the distribution state of all pixels in the processing target area of the image is obtained for each luminance value.

In another embodiment of the present invention, in the judging operation 1b for judging that the image is too bright, a density histogram of a part of the screen (for example, the road part 18 shown in FIG. 5) is obtained, and the luminance value of the histogram is calculated. When the distribution tendency is almost white, it is determined that the image is too bright. In still another embodiment of the present invention, in the determination operation 1c, the average density of a portion (for example, the own vehicle body portion 19 in FIG. 5) that is always in the same state in the input image of the camera 1 is calculated and obtained. If the value has a difference equal to or larger than a predetermined difference compared to the average density during normal running, it is determined that the image is too bright.

In step b4 of FIG. 4, it is determined whether or not the image captured by the camera 1 is too dark due to an unrecognizable condition such as at night. Fig. 7 (1)
Is a diagram for illustrating a state in which an image captured by the camera 1 is too dark at night. FIG. 7A shows an image that is too dark, and FIG. 7B shows a density histogram of the image shown in FIG. 7A. In the judgment operation 2a for determining that the image is too dark, the density histogram of the entire screen is displayed as shown in FIG.
It is obtained as shown in (2), and when the distribution of the luminance value is almost black, it is determined that the image is too dark. That is, in the density histogram, for example, when the number of pixels at a density lower than 51 is equal to or larger than a predetermined value, it is determined that the image is too dark.

In the judgment operation 2b according to another embodiment of the present invention, a density histogram of a part of the screen (for example, the road part 18 in FIG. 5) is obtained. Is determined to be too dark.

In the determination operation 2c according to still another embodiment of the present invention, the average density of a portion (for example, the own vehicle body portion 19 in FIG. 5) of the input image of the camera 1 which is always in the same state is obtained. If it is too dark compared to the average density during normal running, it is determined that the image is too dark.

At step b5 in FIG. 4, the image is
It is determined whether the contrast is low based on fog or the like.
FIG. 8 is a diagram illustrating a state when an image has low contrast. FIG. 8A shows a low-contrast image.
FIG. 8B shows a density histogram of the image. In the determination operation 3a for determining that the image has low contrast,
A density histogram is obtained for the entire screen, and when the tendency of the distribution of luminance values is concentrated in a part, it is determined that the screen has low contrast. For this reason, when the range of the density having the number of pixels equal to or more than the predetermined number in the density histogram is less than the predetermined density value, it is determined that the contrast is low.

In another embodiment of the present invention, in the judging operation 3b for judging that the screen has low contrast, a density histogram of a part of the screen (for example, the road portion 18 in FIG. 5) is obtained, and the distribution of the luminance value is obtained. When the tendency is partially concentrated, it is determined that the contrast is low.

In step b6, it is determined that smear has occurred in the image captured by the camera 1. Such smear occurs under non-recognizable conditions such as backlight. The smear refers to a vertical stripe pattern generated on the screen under the influence of external light of the camera 1 as shown in FIG.

FIG. 9 is a diagram for explaining a state in which smear has occurred in an image picked up by the camera 1. In the image of FIG. 9A, a vertical stripe pattern 20 that is a smear is displayed. The density projection of this image in the vertical direction is shown in FIG.
It is shown in (2). In FIG. 9B, the horizontal axis represents the horizontal position of the screen, and the vertical axis represents the sum of the densities of the respective pixels in the vertical direction. In the judging operation 4a for judging the occurrence of smear, the density projection in the vertical direction of the entire image is obtained, and when a part of the projection value protrudes as indicated by a reference numeral 21, the smear is generated. Judge.

FIG. 10 is a diagram for explaining a specific operation of the judging operation 4a in which smear has occurred in step b2 of FIG. The process proceeds from step c1 to step c2. In the density projection in the vertical direction of the entire image shown in FIG. 9 (2), first, for each pixel position in the horizontal direction (horizontal direction in FIG. 9 (2)), It is determined whether or not one peak p1 exists, that is, whether or not the position has a higher density sum than the density sums on both sides. In the next step c3, a range of ± L on both sides of the first peak p1 is set. This ± L
Is, for example, a range of ± 10 pixels on both sides of the position of the own pixel. In step c4, the second peak p2 higher or lower than the first peak p1 in this range ± L
Ask for. In step c5, the first and second peaks p
The difference ΔP between 1 and p2 (= │p1−p2│) is calculated and obtained. In step c6, the difference Δp is set to a predetermined value Δp1.
It is determined whether this is the case (Δp ≧ Δp1). Difference Δp
Is greater than or equal to a predetermined value Δp1 and the step c7
Then, it is determined that smear has occurred.

In another embodiment of the present invention, in a judging operation 4b for judging occurrence of smear, a density projection in a vertical direction of a part of the screen (for example, the road portion 18 in FIG. 5) is obtained, and the projection value of the projection value is obtained. When a part is protruding, it is determined that smear has occurred.

At step b7 in FIG. 4, it is determined that the screen is blurred due to unrecognizable conditions such as fog, camera abnormality, and dirt on the windshield 10, that is, blurred.

FIG. 11 is a diagram for explaining a state in which the screen is blurred, and FIG. 12 is a diagram for explaining a state in which the screen of the same subject is not blurred in normal light. FIG. 11B shows the density projection in the vertical direction of the entire screen of the blurred screen shown in FIG. 11A. FIG. 12 (2) shows the density projection in the vertical direction of the entire screen of the clear screen of FIG. 12 (1). FIG. 11 (2)
Indicates the vertical sum of edge values corresponding to each pixel position in the horizontal direction of the screen, and the same applies to FIG. 12 (2). In the operation of determining whether or not the screen is blurred, an edge extraction technique is executed. The edge value is a value obtained by extracting a portion in the image where the luminance value changes abruptly. By performing edge extraction, it is possible to extract a contour of an object in the image.

FIG. 13 is a diagram showing operation coefficients for extracting an edge value. The edge value of the coordinates (i, j) is Δf
If (i, j) and the density value are f (i, j), Δf (i, j) = f (i−1, j−1) +2 f (i−1, j) + f (i−1, j) j + 1) -f (i + 1, j-1) -2f (i + 1, j) -f (i + 1, j + 1) (1) In the judgment operation 5a that the screen is blurred, FIG. 11 (2) and FIG. The edge image shown in 12 (2) is obtained, and when an edge value equal to or more than a predetermined number is not obtained, it is determined that the screen is blurred. In another embodiment of the present invention, in a determination operation 5b of determining that the screen is blurred, an edge image of a part of the screen (for example, the road portion 18 in FIG. 5) is obtained, and a predetermined number or more of edge values are obtained. If not, judge that the screen is blurred.
In still another embodiment of the present invention, in the determining operation 5c for determining that the screen is blurred, an edge of a portion where a contour is always obtained (for example, a boundary portion between the vehicle body portion 19 and the road in FIG. 5). The value is obtained, and when a predetermined number or more of edge values cannot be obtained, it is determined that the screen is blurred.

FIG. 14 shows that in step b8 of FIG.
FIG. 3 is a diagram for explaining a state in which an image from the camera 1 is blurred due to a non-recognizable condition such as rain adhering to the outer surface of the windshield 10. FIG. 14 (1)
Then, the blurred image is shown. In FIG. 14 (2),
FIG. 15 illustrates edge projection in the vertical direction of the image in FIG. When the image is blurred, the determining operation 6a determines that the image is blurred when the length and direction of the edge of the recognized target object are inappropriately distorted. The length of the edge is obtained when a length L1 in which the predetermined sum of edges in FIG. 14 (2) is equal to or larger than a predetermined value is obtained.
It may be determined that the image is blurred.

In step b9 in FIG. 4, when a phenomenon occurs in which the image picked up by the camera 1 does not change even though the vehicle is moving, it is recognized that the camera is abnormal due to a breakdown or the like. A determination operation 7a for determining the impossible condition is performed. In this judging operation 7a, when an output from the detecting means for detecting the wheel speed of the vehicle body 6 is obtained, and when the images on the plurality of screens are not sequentially changed, it is determined that the images have not changed.

In another embodiment of the present invention, in Table 2, in the judging operation 7b in which it is judged that the image has not changed even though the vehicle is moving, the difference between the entire image and the previous input image is determined. Is determined, and when a difference value equal to or more than a predetermined number is not obtained, it is determined that the image has not changed. The difference is a value obtained by calculating a difference between luminance values of corresponding pixels of two images temporally adjacent to each other. If there is no or little change in the image, the difference value becomes zero or a small value. Become. In this way, the difference between the luminance values of the corresponding pixels between the previous input image and the current image at this time is obtained, and the level is discriminated, whereby it can be determined that the image has not changed.

In still another embodiment of the present invention, in Table 2, a part of the screen (for example, the road shown in FIG. 5) is used in the judgment operation 7c for judging that the image has not changed even though the vehicle is moving. The difference between the previous input image of the portion 18) and the current input image is obtained as described above, and if no difference value equal to or more than a predetermined number is obtained, it is determined that the image has not changed.

As other image phenomena shown in Table 2, in the judging operation 8a for judging an unrecognizable condition which is an abnormality caused by a camera failure or the like, the horizontal or horizontal image included in the image signal obtained from the camera 1 is determined. The level of the vertical synchronization signal is abnormal, the level of the video signal in the image signal is abnormal in the determination operation 8b, and the level of the video signal is abnormal in the determination operation 8c.
When the interval, that is, the period of the synchronization signal in the image signal is abnormal, it is determined that an unrecognizable condition that the camera is abnormal has occurred. The image signal is a composite video signal including a video signal and horizontal and vertical synchronization signals. NTSC (Nation
al Television Standard Committee), the white level and the upper and lower values of the sync signal are determined. The interval between the vertical sync signals is 16.7 msec, and the interval between the horizontal sync signals is 63.5 μsec. Stipulated.

When the phenomena 1 to 8 of the image shown in Table 2 are detected in steps b3 to b10 in FIG. 4, the condition that the recognition processing is impossible is notified by the notifying means 12 in step a9 in FIG. Is done.

FIG. 15 is a block diagram showing the overall configuration of another embodiment of the present invention. This embodiment is similar to the embodiment of FIGS. 1 to 14 described above, and corresponding parts are denoted by the same reference numerals. In this embodiment, after the phenomenon of the image in Table 2 in the above-described embodiment is determined and the unrecognizable condition is notified, it is not immediately determined that the image cannot be recognized. Or the image data from the camera is corrected so that image recognition processing can be performed. In FIG. 15, a driving circuit of the wiper 11 for wiping the outer surface of the windshield 10 is controlled by the processing circuit 2. An analog signal of image data from the camera 1 is converted into a digital signal in an analog / digital converter 23,
In 4, the same arithmetic processing operation as described above is performed. The light receiving amount adjusting means 25 of the camera 1 adjusts the aperture and / or shutter speed of the camera 1 in response to a control signal from the processing means 2. Other configurations and operations are the same as those of the above-described embodiment.

FIG. 16 is a flowchart for explaining the operation of the processing circuit main body 24 of the embodiment shown in FIG. Steps d2, d4 to d8 in FIG. 16 correspond to steps a2, a5 to a9 in FIG.
Is the same as Move from step d1 to step d2,
Image data of camera 1 is analog / digital (A / D)
The signal is supplied to the processing circuit main body 24 via the converter 23. In step d3, a determination is made that recognition is not possible, and if recognition is not possible, the input image data is corrected. After the correction, the recognition process is performed to make the image recognizable, and when the recognition process is not possible even after such correction, the region where the recognition process cannot be performed may be excluded from the recognition process region. Good. In step d4, the obtained corrected image data is subjected to recognition processing, for example, recognition processing of an object such as the white line on a road. If no unrecognizable condition exists, step d2
The image data from the camera 1 given in step
In step 4, the recognition processing is performed as it is. Other operations are
This is the same as the above-described embodiment.

FIG. 17 is a flowchart for explaining a more specific operation of step d3 in FIG. 16 in the embodiment shown in FIGS. In steps e1 to e5, the conditions of the image phenomena and the image non-recognition processing disabled conditions 1 to 4 and 6 shown in Table 3 are executed by the determination operation of Table 2. These image phenomena and unrecognizable conditions 1 to 4 and 6 are the same as the reference signs of the image phenomena and unrecognizable conditions shown in Table 2.

[0050]

[Table 3]

When it is determined in step e1 in FIG. 17 that the screen is too bright, and when it is determined in step e2 that the screen is too dark, the process proceeds to step e6.
It is determined whether the flag a is logic “1”. In the initial state, the flag a is logic “0”. Then, the process proceeds from step e6 to step e7, where the flag a is set to logic “1”.
In step e8, the aperture and / or shutter speed provided in the camera 1 is adjusted, and the state is adjusted so that image recognition is possible. Thereafter, steps e1 and e2 are performed again.
Return to When the image is too bright or the image is too dark despite the aperture and / or shutter speed being adjusted, the process again goes to e6, and since the flag a is logic “1”, the next step e9 Then, the region having the unrecognizable condition is excluded from the entire recognition processing region of the image. At step e10, the flag a is returned to logic "0". Thus, the object recognition process is performed in step d4 in FIG.

When it is determined in step e3 in FIG. 17 that the contrast of the image is low, the process proceeds from step e11 to step e12, where the flag b is set to logic "1". In the next step e13, the processing means 2 is provided. The sensitivity of the analog / digital converter 23 is increased. That is, in the analog / digital converter 23, adjustment is performed so that the level difference between the input analog signals represented by the adjacent digital signals is increased and the contrast is increased.
After step e13, the process returns to steps e1 to e3 again, and proceeds from step e11 to step e14. If the screen is still low contrast, step e14
Then, the area having the unrecognizable condition is excluded from the recognition processing area of the entire image, and in the next step e15, the flag b is returned to logic "0". Thus, step d in FIG.
In step 4, an object recognition process is performed.

When it is determined in step e4 that image smearing has occurred, the operation proceeds to step e6. When the occurrence of smear is detected in step e4, a control signal is transmitted to the camera 1, and the aperture and / or shutter speed of the lens are adjusted and changed so that smear does not occur. When smear occurs in this image, when it cannot be excluded by the above adjustment operation,
When performing image recognition processing, an area where smear has occurred is excluded from the recognition processing area. As a result, the recognition process can be performed even when smear occurs.

In step e5, the windshield 10
When it is determined that an unrecognizable condition that raindrops have adhered to step e16, the process proceeds from step e16 to step e17, and the flag c is set to logic "1". In step e18, the wiper 11 is driven, and the outer surface of the windshield 10 is wiped and cleaned. Returning to step e5 again, if the unrecognizable condition that raindrops are attached is determined even though the wiper 11 is driven, step e1 is performed.
Then, the process proceeds from step 6 to step e19, in which the area where the unrecognizable condition exists is excluded from the entire recognition processing area of the image. At step e20, the flag c is returned to logic "0". Thereafter, an object recognition process is executed in step d4 of FIG.

FIG. 18 is a block diagram showing an overall configuration of still another embodiment of the present invention. This embodiment is similar to the above-described embodiment, and corresponding parts are denoted by the same reference numerals. In this embodiment, various sensors indicated by reference numerals 27 to 34 in FIG. 18 are used to determine the surrounding environment which is a condition in which recognition processing using the image data of the camera 1 is impossible. Table 4 shows the conditions that cannot be recognized by each sensor.

[0056]

[Table 4]

FIG. 19 is a flowchart for explaining the operation of processing circuit 2 shown in FIG. This figure 1
In step f1 of step 9, when the sensor 27 for detecting the operation of the wiper detects that the operation of the wiper is not performed, or when the raindrop sensor 28 provided on the vehicle body 6 detects the rainfall state, the recognition process is not performed. A signal representing a possible condition is derived.

In step f2, the friction coefficient μ of the road surface is detected by the μ sensor 29, and the friction coefficient μ of the road is detected.
However, when detecting an unrecognizable condition that is small, that is, a state of rain, a state of rain, or a state of snow, a signal representing the unrecognizable condition is derived. Accordingly, the notifying unit 12 displays and notifies, for example, that "the white line of the road cannot be detected due to poor road condition".

In step f3, the sound sensor 30 is used.
The sound of a raindrop colliding with the vehicle body 6 is detected. When the sound of the raindrop is detected by the sound sensor 30, a signal representing an unrecognizable condition is derived. In step f4, fog is generated from the altitude, humidity and temperature detected by the sensor 31 including the altimeter, hygrometer, and thermometer provided on the vehicle body 6, and represents a condition that the recognition process is not possible. Derive the signal.

In step f8, when it is determined in step f4 that fog, which is an unrecognizable condition, is generated, in step f8, image data from the camera 1 is further processed, and As in the case of the embodiment, the condition for which the recognition process is not possible is detected and determined. When it is determined that the unrecognizable condition due to the arithmetic processing of the image data obtained in step f8 is fog, and therefore matches the fog which is the unrecognizable condition detected by the output of the sensor in step f4, The not-recognizable condition is reported by the reporting means 12 in step f9. When each of the detected conditions does not match in step f8, the process proceeds to the next step f5 to perform the recognition process.

In another embodiment of the present invention, step f
8 may be omitted, and when fog, which is an unrecognizable condition, is detected in step f4, the recognition process may not be performed, and notification may be performed in step f9.

In step f5, the illuminance meter 32 detects an illuminance value, and if it is detected that the condition is backlight or true darkness, which is an unrecognizable condition, the recognition process is not performed and the illuminance value is not recognized. The notification means 12 notifies that the light is backlight or true dark. Instead of the illuminometer, a solar sensor having high directivity may be used to detect the solar radiation intensity. With this solar radiation sensor, it is possible to detect a non-recognizable condition that is backlight or darkness.

In step f6, the direction of the sun is estimated by the time and direction sensor 33 of the clock, and when it is determined that the camera 1 is in backlight, the unrecognizable condition is notified by the notification means 12. , Does not perform recognition processing.

In step f7, G for detecting the current position of the vehicle provided in the navigation device (abbreviated as "navigation").
The state of satellite reception from a PS (global positioning system) device is detected by the sensor 34, and when reception is not possible, the fact that the vehicle is traveling in a tunnel, which is an unrecognizable condition, is reported by the reporting means 12 and recognition processing is not performed. . This GPS device is indicated by reference numeral 34 in FIG. In this manner, the navigation system displays the route of the vehicle to reach the destination input in advance on the two-dimensional display surface, and displays the current position of the vehicle on the map displayed on the two-dimensional display surface. Since the image non-recognizable condition is determined using the output of the GPS device in such a navigation device, the configuration for determining the non-recognizable condition can be simplified. Instead of the navigation device, a part of the configuration of other various devices may be shared for detecting the unrecognizable condition.

The image recognition apparatus according to the present invention may be mounted on a vehicle such as an automobile, or may be mounted on a vehicle without being mounted on a vehicle.

[0066]

According to the first aspect of the present invention, when recognition processing of image data from a camera by the recognition processing means is impossible, a condition that is not possible, that is, a cause is detected by the condition detection means. The notification is made by the notification means, so that the operator can determine whether the image input to the camera is abnormal or inappropriate, and thus the reliability of the image recognition device can be ensured. it can.

According to the second aspect of the present invention, when the condition detecting means detects and determines that the condition that the recognition processing is not possible, the correcting means corrects the input image to the camera, or The image data obtained from the image processing apparatus can be corrected, thereby enabling the recognition processing unit to perform the recognition processing of the target object. Therefore, it is possible to enlarge the use area that is set as the corresponding environment and improve the performance of the image recognition apparatus.

According to the third aspect of the present invention, the condition detecting means determines whether or not the recognition processing of the object by the recognition processing means is impossible based on the output from the sensor for detecting the surrounding environment. As a result, it is not necessary to judge whether or not recognition is possible by arithmetic processing of the image data from the camera, the arithmetic processing can be reduced, and the detection and determination of the condition in which the recognition processing by the condition detecting means is not possible can be performed. Can be done quickly.

According to the fourth aspect of the present invention, the condition detected by the output of the sensor and the condition that the recognition processing means cannot perform the recognition processing detected by performing arithmetic processing on the image data from the camera. However, when a match is found, the matching condition is output by the notifying means, so that it is possible to determine with high accuracy that recognition is not possible, and the reliability of the image recognition device can be further improved.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an entire configuration of a driving support and automatic driving device including an image recognition device according to an embodiment of the present invention.

FIG. 2 is a simplified side view of a vehicle body 6 of an automobile.

FIG. 3 is a flowchart for explaining the operation of the processing means 2;

FIG. 4 is a flowchart showing a part of the operation of the processing means 2 shown in FIG. 1 in further detail.

FIG. 5 is a front view showing an image 16 captured by the camera 1 and displayed on a two-dimensional display surface of the notifying unit 12.

FIG. 6 is a diagram showing a state in which an image captured by the camera 1 is too bright.

FIG. 7 is a diagram illustrating a state in which an image captured by the camera 1 is too dark at night.

FIG. 8 is a diagram illustrating a state when an image has low contrast.

FIG. 9 is a diagram for explaining a state in which smear has occurred in an image captured by the camera 1.

FIG. 10 is a diagram for explaining a specific operation of a determination operation 4a in which a smear has occurred in step b2 of FIG. 4;

FIG. 11 is a diagram for explaining a state in which the screen is blurred.

FIG. 12 is a diagram for explaining a state where the screen of the same subject is clear and not blurred in normal light.

FIG. 13 is a diagram showing operation coefficients for extracting an edge value.

FIG. 14 is a diagram for explaining a blurred image.

FIG. 15 is a block diagram showing the overall configuration of another embodiment of the present invention.

FIG. 16 is a flowchart for explaining the operation of the processing circuit main body 24 of the embodiment shown in FIG.

FIG. 17 is a flowchart for explaining a more specific operation of step d3 in FIG. 16 in the embodiment shown in FIGS. 15 and 16;

FIG. 18 is a block diagram showing an overall configuration of still another embodiment of the present invention.

FIG. 19 is a flowchart for explaining the operation of the processing circuit 2 shown in FIG. 18;

FIG. 20 is a flowchart for explaining the operation of the prior art.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Camera 2 Processing circuit 3 Control circuit 10 Windshield 11 Wiper 12 Notification means 16 Image 23 Analog / Digital converter 24 Processing circuit main body 25 Light receiving amount adjusting means 32 Illuminance meter 33 Direction sensor

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Masahiro Babasaki 1-2-28, Goshodori, Hyogo-ku, Kobe City, Hyogo Prefecture Inside Fujitsu Ten Co., Ltd. (72) Inventor Akihiro Ota 1 Goshodori, Hyogo-ku, Kobe City, Hyogo Prefecture Fujitsu Ten Co., Ltd. (72) Inventor Kenji Oka 1-2-28 Goshodori, Hyogo-ku, Kobe City, Hyogo Prefecture Inside Fujitsu Ten Co., Ltd. (72) Inventor Keiji Kunitomo Hyogo-ku, Hyogo Prefecture Goshodori 1-2-28 Fujitsu Ten Limited F term (reference) 5H180 AA01 BB17 CC04 CC24 LL01 LL02 LL09 5L096 BA04 CA02 DA03 FA06 FA32 FA37 FA38 GA08

Claims (4)

[Claims] 1. A camera, a condition detecting means for performing arithmetic processing on image data from the camera and detecting a condition in which recognition processing is impossible, and responding to an output of the condition detecting means to disable recognition processing. An image recognizing apparatus comprising: a notifying unit for notifying a condition; and a recognition processing unit for responding to an output of the condition detecting unit and performing a recognition process of image data from a camera when a recognition process is possible. 2. A camera, a condition detecting means for performing arithmetic processing on image data from the camera and detecting a condition in which recognition processing is impossible, and responding to an output of the condition detecting means to correct an image input to the camera. Or correction means for correcting image data from the camera, and recognition processing means for recognizing image data from the camera in which the input image has been corrected by the correction means or image data corrected by the correction means. An image recognition device characterized by the above-mentioned. 3. A camera, a sensor for detecting a surrounding environment of the camera, recognition processing means for performing recognition processing of image data from the camera, and image data from the camera responsive to the output of the sensor. An image recognition apparatus comprising: condition detecting means for detecting a condition in which recognition processing cannot be performed; and notifying means for responding to an output of the condition detecting means and reporting a condition in which recognition processing cannot be performed. . 4. The condition detecting means, when the condition detected by the output of the sensor and the condition detected by performing arithmetic processing on the image data from the camera match,
4. The image recognition apparatus according to claim 3, wherein an output representing the condition is provided to a notification unit.