JP2001211466A - Image processing system having self-diagnostic function - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a self-diagnostic function that recognizes malfunction of the eye of the system itself like in a human eye with a relatively simple image processing configuration at a relatively low cost even in the case of mounting a monitor camera having a plurality of image pickup elements on a traveling object. SOLUTION: This image processing system having the self-diagnostic function is provided with a plurality of image pick-up means (a pair of left and right cameras 1a and 1b for a stereo camera) which image the surrounding of a vehicle (traveling object) from a monitoring position on the vehicle and obtain a video signal, a monitor means (correlation operation and distance operation circuit 23) which monitors the surrounding of the vehicle by using each video signal obtained by the image pick-up means, and a self-diagnosing means (characteristic quantity extraction circuit (I) 25a, (II) 25b and an integrated decision circuit 26) which relatively compares frame images of the same time of respective image signals obtained by the plurality of image pick-up means when the monitor means performs monitoring and performs self-diagnosis of the soundness of the plurality of image pick-up means in accordance with the comparison results.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing system having a self-diagnosis function and, more particularly, to a device for performing self-diagnosis by comparing a plurality of images obtained by a stereo camera or the like.

[0002]

2. Description of the Related Art With the advancement of elemental technologies such as an imaging system and an image processing system in recent years, for example, a CCD camera or the like is mounted on a vehicle as one of driving support systems for a vehicle, and a camera mounted on the vehicle is used. There is an increasing demand for a device that processes an obtained image by image processing or the like, thereby monitoring obstacles around the vehicle and other vehicles.

As a conventional example of such a monitoring apparatus using an on-vehicle camera and an apparatus / method related thereto, for example, an obstacle detecting apparatus for a vehicle (for example, Japanese Patent Laid-Open No. 6-273170).
Publication), a vehicle recognition device (for example, JP-A-7-3348)
No. 00), a vehicle exterior monitoring device (for example,
No. 83461, JP-A-10-283462, JP-A-1
Japanese Patent Application Laid-Open Nos. 0-283377, 10-285558, 11-39596, and 11-21138), a vehicle periphery monitoring device, and an obstacle detection method using the same (for example, No. 117339, JP-A-10-117341, JP-A-10-117342
No.) etc.

Japanese Patent Application Laid-Open No. Hei 6-273170 discloses an obstacle detecting device for a vehicle that identifies the distance to an obstacle from a captured image input by a pair of left and right image sensors for imaging the periphery of the vehicle. Has been described.

Japanese Patent Application Laid-Open No. 7-334800 discloses a vehicle recognition device that recognizes a preceding vehicle from a road image input using an on-vehicle camera, and thereby measures an inter-vehicle distance with the vehicle.

[0006]

In the above-described apparatus using the on-vehicle camera, when monitoring an object or the like around the vehicle, it is not usually necessary for a human to look at an image to make a judgment, but to obtain an image using an imaging system. It is assumed that the image is processed by an image processing system such as image recognition. Therefore, in each of the above-described conventional apparatuses, a failure of an imaging system or an image processing system occurs, or image recognition or the like is erroneously executed due to fogging of the lens of the imaging system or dirt. However, there is a problem that even if an inconvenient situation occurs in the device, the fact cannot be grasped by a human.

As a countermeasure against such a problem, for example, Japanese Patent Application Laid-Open No. H11-126248 proposes an image processing system in which a failure of an image sensor or an inconvenient situation in image processing is diagnosed and detected in the system. ing. This image processing system compares a measured value of an elapsed time of a synchronization signal separated from an image signal, a measured value of luminance of an image signal in one screen, an amplitude hold value of an image signal, and the like with a preset set value. Then, an abnormality is determined in accordance with the comparison result, and when it is determined that the abnormality is abnormal, an alarm is issued.

However, in this system, a synchronization signal, luminance information, an amplitude hold value, and the like are measured for each image sensor, and an abnormality is determined by comparing the measured value with a set value. Only a short circuit is detected, and the configuration of a signal processing circuit for image processing or the like becomes extremely complicated, and the system may be expensive.
In particular, a problem is supposed that this tendency becomes remarkable in proportion to the number of imaging elements.

The present invention has been made in view of such a conventional problem, and has a relatively simple image processing configuration even when a monitoring camera having a plurality of image sensors is mounted on a moving body. It is an object of the present invention to realize a self-diagnosis function for recognizing a defect of an eye of the system itself like a human eye at a relatively low cost.

[0010]

SUMMARY OF THE INVENTION In order to achieve the above object, the present invention focuses on a means for relatively checking the soundness of a system by comparing a plurality of camera images as a self-diagnosis function. .

In this self-diagnosis function, for example, two or more cameras are installed on a vehicle as cameras for monitoring the peripheral portion thereof at a predetermined distance, and a plurality of camera images taken at the same time by each camera are used. Calculates the feature value, compares the calculated feature value for each of the cameras, and notifies the driver etc. that the surveillance system is incompatible if the difference in the size of the feature value exceeds a certain value. It is possible to

Here, as a method of calculating the feature amount, the feature amount is calculated by performing image processing such as a process of calculating an optical flow and a process of binarizing a differential processing value with reference to a threshold value. The optical flow means a light vector representing the movement of a minute area on the screen, and by calculating this, for example, a certain moving object can be detected from another moving object.

An image processing system having a self-diagnosis function according to the present invention has been completed based on the above-mentioned point of interest, and obtains a video signal by imaging the periphery of a moving body from a monitoring position on the moving body. A plurality of imaging means; a monitoring means for monitoring the periphery of the moving body using each video signal obtained by the plurality of imaging means; and a monitoring means for monitoring by the plurality of imaging means when monitoring by the monitoring means is performed. And self-diagnosis means for performing self-diagnosis of the soundness of the plurality of imaging means in accordance with the comparison result.

[0014] The self-diagnosis means individually determines a predetermined characteristic amount for each of the plurality of imaging means from frame images of a current time and a previous time with respect to a frame image of each video signal obtained by the plurality of imaging means. Comparing the feature amounts calculated for each of the plurality of imaging units by the feature amount calculation unit, and when the coincidence between the feature amounts is lower than a predetermined value, It is possible to provide a determining means for determining that the means is defective.

The feature amount calculating means includes means for calculating an optical flow from the current and previous frame images, and a differential processing amount based on a threshold value based on the current and previous frame images. And means for calculating the binarization ratio can be provided.

[0016] The determination means may include means for notifying that the imaging means is defective when the degree of coincidence between the respective feature amounts is lower than a predetermined value.

As a preferred example, the moving body is a vehicle, the plurality of image pickup units include a pair of left and right image pickup devices functioning as a stereo camera, and the monitoring unit is obtained by the plurality of image pickup units. Means for calculating a distance to an object in a field of view around the moving object based on the respective video signals.

[0018]

An embodiment of an image processing system having a self-diagnosis function according to the present invention will be described below with reference to the drawings.

The image processing system shown in FIG. 1 is applied to a monitoring system that measures the distance to an obstacle by forward monitoring. The image processing system 1 captures an object such as an obstacle as an eye for forward monitoring. An image processing unit 2 that measures the distance to the target object based on the image data from the imaging system 1 and that performs image processing for self-diagnosing a defect / soundness of the system itself.

The imaging system 1 may have a plurality of imaging devices capable of imaging the same object from a plurality of different viewpoints. In this embodiment, a pair of right and left right cameras 1a functioning as a stereo camera And left camera 1b
including.

The cameras 1a and 1b are, for example, CCDs
As shown in FIG. 2, they are arranged side by side at a fixed distance from each other at a monitoring position on the vehicle front side where a field of view for forward monitoring can be secured. The right camera 1a captures a right image of the stereo image and sends the video (video) signal (I) to the image processing unit 2, while the left camera 1b captures a left image of the stereo image. The video signal (II) is sent to the image processing unit 2.

The image processing unit 2 includes a timing signal generation circuit 21 for generating a timing signal for controlling the operation timing of image processing from the image pickup by the system, and a timing signal from the timing signal generation circuit 21 for synchronizing each signal. Video signals (I) from the cameras 1a and 1b,
Frame memory circuits (I) 22a and (II) 22b each having a built-in frame memory for storing (II) as left and right frame images (I) and (II) in time series, and both frame memory circuits (I) 22a; (II) Correlation calculation / distance calculation circuit 2 for inputting left and right frame images from 22b
3 and a self-diagnosis system 24.

The correlation calculation / distance calculation circuit 23 synchronizes with the timing signal from the timing signal generation circuit 21 while synchronizing with both frame memory circuits (I) 22a and (II) 22.
b) performs a correlation calculation and a distance calculation based on a predetermined processing algorithm for measuring a distance to an object based on the left and right frame images (I) and (II) captured at the same time from b. Send to diagnostic system 24.

The self-diagnosis system 24 has a function of comparing the left and right frame images (I) and (II) taken at the same time from the two frame memory circuits (I) 22a and (II) 22b with the previous and current times. Feature extraction circuits (I) 25a and (II) 25b for extracting a predetermined feature from the frame image of
And a comprehensive determination circuit 26 that comprehensively determines the failure / soundness of the system itself based on the feature values from the circuits (I) 25a and (II) 25b.

The feature amount extraction circuits (I) 25a and (II) 2
5b executes, for example, a processing algorithm for calculating an optical flow as a feature amount from the previous and current frame images, and sends the calculation result (feature amount) of the optical flow to the comprehensive judgment circuit 26.

The comprehensive judgment circuit 26 judges the degree of coincidence between the two characteristic quantities from, for example, the characteristic quantity extraction circuits (I) 25a and (II) 25b, whereby the coincidence (%) between the two characteristic quantities is equal to or more than a predetermined value. If it is determined that the correlation is within the range that allows the soundness of the system itself, the calculation result (image recognition result) from the correlation calculation / distance calculation circuit 23 is output. If it is determined that the value is smaller than the range (that is, it is not within the range that allows the soundness of the system itself, that is, it is defective), a processing algorithm that outputs an alarm signal to the imaging system 1 is executed.

FIG. 3 shows an example (steps S1 to S10) of a processing procedure of the image processing system according to this embodiment.

In FIG. 3, the image processing system comprises:
When starting the system, objects around the vehicle (objects in the field of view)
The video signal (I) forming a right image and the video signal (I) forming a left image by both cameras 1a and 1b of the imaging system 1.
I) are obtained (steps S1a and S1b), and the video signals (I) and (II) are transferred to both frame memory circuits (I) 22a and (II) 22b of the image processing unit 2 by the left and right frame images ( I) and (II) are stored in chronological order (steps S2a and S2b).

In this image processing system, the correlation calculation / distance calculation circuit 23 converts the left and right frame images (I) and (II) stored in the two frame memory circuits (I) 22a and (II) 22b. The distance to the object within the field of view is calculated (step S3), while the processing by the self-diagnosis system 24 is executed in parallel with this (step S4).

In the processing of step S4, the characteristic amount extraction circuits (I) 25a and (II) 25b use the left and right frame images (I) and (II) for the optical frame The flow is calculated individually (steps S5a and S5b), and the calculation results (feature amounts) A and B are received and the process by the comprehensive judgment circuit 26 is executed (step S6).

In the process of step S6, the degree of coincidence between the calculation results (feature amounts) A and B of the optical flows of the left and right frame images (I) and (II) by the comprehensive judgment circuit 26 is equal to or more than α%. Is determined (step S
7), YES in this determination (coincidence degree of both feature amounts A and B is α)
%), The gate to which the distance calculation result (recognition result) by the correlation calculation / distance calculation circuit 23 is input is turned on and the recognition result is output (steps S8 and S).
9) On the other hand, if the determination in step S7 is NO (both feature amounts A,
If the degree of coincidence of B is not equal to or more than α%), it is output that some abnormality has occurred in the imaging system 1 (step S10).

Therefore, according to this image processing system,
The system is configured to relatively check the soundness of the system by comparing multiple camera images, so even if a failure of the image sensor etc. occurs, a monitoring camera with multiple image sensors is mounted on the moving body However, even in the case where a self-diagnosis function for recognizing a defect in the eyes of the system itself like a human eye can be realized relatively inexpensively with a relatively simple image processing configuration, thereby increasing the reliability of the system. Can be further enhanced.

In this embodiment, an algorithm for calculating an optical flow is used as a processing algorithm for extracting a feature value, but the present invention is not limited to this. For example, step S5a shown in FIG.
Steps S51a and S51 shown in FIG.
It is also possible to adopt each processing of 51b. This figure 4
In the processing of steps S51a and S51b shown in (1), the previous and current frame images are differentiated as a processing algorithm for extracting a feature amount, and the processing result is binarized based on a preset threshold value. The binarization ratio is obtained.

[0034]

As described above, according to the present invention, the system is configured to relatively check the soundness of the system by comparing a plurality of camera images. Even when a camera is mounted, a relatively simple image processing configuration realizes a self-diagnosis function that recognizes a defect in the eyes of the system itself (such as a failure in an image sensor) like a human eye at a relatively low cost. This can further increase the reliability of the system. This effect can be maximized by applying it to a device that supports driving by mounting a CCD camera or the like on a car and performing image processing on the image.

[Brief description of the drawings]

FIG. 1 is a schematic diagram of an image processing system having a self-diagnosis function according to an embodiment of the present invention.

FIG. 2 is a schematic block diagram illustrating the overall configuration of an image processing system having a self-diagnosis function.

FIG. 3 is a schematic flowchart illustrating a processing procedure of the image processing system.

FIG. 4 is a schematic flowchart showing another feature amount calculation process.

[Explanation of symbols]

 Reference Signs List 1 imaging system 1a right camera 1b left camera 2 image processing unit 21 timing signal generation circuit 22a frame memory circuit (I) 22b frame memory circuit (II) 23 correlation calculation / distance calculation circuit 24 autocorrelation system 25a feature amount extraction circuit (I ) 25b Feature extraction circuit (II) 26 Comprehensive judgment circuit

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G08G 1/097 H04N 5/225 C 5H223 H04N 5/225 7/18 F 9A001 7/18 13/02 13/02 02 G06F 15/62 380 F term (reference) 5B057 AA06 BA19 BA23 CF03 DA03 DA15 DA16 DC03 DC05 DC08 DC09 5C022 AA04 AB61 AB68 AC11 AC13 AC18 AC69 5C054 AA02 CC02 CH02 EA01 EA05 FC11 FE28 FF05 5C061 AB04 CC08 AA01 BB18 CC04 LL01 LL02 LL04 LL06 5H223 AA10 BB09 DD03 FF03 9A001 HH23 KZ54 LL06

Claims (7)

[Claims] A plurality of image pickup means for picking up an image of a periphery of a moving object from a monitoring position on the moving object to obtain a video signal, and using the respective video signals obtained by the plurality of image pickup means, Monitoring means for monitoring the surroundings, and when monitoring by the monitoring means is performed, frame images at the same time of each video signal obtained by the plurality of imaging means are relatively compared with each other, and according to the comparison result, An image processing system having a self-diagnosis function, comprising: self-diagnosis means for self-diagnosing the soundness of a plurality of imaging means. 2. The invention according to claim 1, wherein said self-diagnosis means determines a predetermined characteristic of a frame image of each video signal obtained by said plurality of imaging means from a current frame image and a previous frame image. A feature amount calculating means for individually calculating an amount for each of the plurality of imaging means; and a feature amount calculated for each of the plurality of imaging means by the feature amount calculating means. An image processing system having a self-diagnosis function, comprising: a determination unit that determines that the imaging unit is defective when is smaller than a predetermined value. 3. The invention according to claim 2, wherein the feature amount calculating means calculates an optical flow from the current and previous frame images and a differential processing from the current and previous frame images. An image processing system having a self-diagnosis function, comprising: at least one of means for binarizing an amount based on a threshold value and calculating a ratio of the binarization. 4. The invention according to claim 2, wherein the determination means includes means for notifying that the imaging means is defective when the degree of coincidence between the feature amounts is lower than a predetermined value. An image processing system having a self-diagnosis function. 5. The image processing system according to claim 1, wherein the moving body is a vehicle. 6. The self-diagnosis function according to claim 1, wherein said plurality of image pickup means have a pair of left and right image pickup devices functioning as a stereo camera. An image processing system having: 7. The apparatus according to claim 1, wherein the monitoring unit is configured to detect an object within a visual field around the moving body based on each video signal obtained by the plurality of imaging units. An image processing system having a self-diagnosis function, comprising means for calculating a distance.