JP2002354463A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem of a prior art image processor that a fixed pattern noise where the luminance of a specific pixel undergoes halation or black skip when the temperature of an imaging element is high or low and reliability of image processed recognition results deteriorates greatly. SOLUTION: An image picked up by an imaging means 1 is stored in an image storage means 2 and a recognizing/image processing means 3 reads out the image and extracts an object. Recognition results including the extracted object is transmitted to a control unit by a communication means 5 and a temperature measuring means 4 measures the temperature of the imaging means 1. Based on the measured temperature, a signal for prohibiting use of the recognition results being transmitted by the communication means 5 is delivered, thus preventing erroneous operation of the control unit.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for providing control or warning information for real-time object recognition.

[0002]

2. Description of the Related Art Using an image pickup means of a vehicle-mounted camera,
As described in Japanese Patent Application Laid-Open No. 2001-33238, as an optical object recognizing device for recognizing an object outside a vehicle, the validity of an image captured based on a luminance value of a region where a part of a light receiving unit is shielded from light is described. There is something to judge. Further, as a camera with a temperature detecting device, Japanese Patent Application Laid-Open No. 2000-34728
As described in Japanese Patent Application Publication No. 4 (1999) -2004, there is a device that prohibits photographing at a low temperature or a high temperature at which a photographing operation relating to a mechanical component is not guaranteed.

[0003]

Problems to be Solved by the Invention JP-A-2001-332
In the object recognition device disclosed in Japanese Patent No. 38, it is necessary to scan the entire light-shielded area for image processing when determining the validity of an image, and if it is used for real-time recognition, it is disadvantageous in processing time. On the other hand, when the temperature of the image sensor becomes high or low, fixed pattern noise occurs in which the luminance value of a specific pixel becomes overexposed or overexposed, and the reliability of the recognition result obtained by performing image processing based on the luminance distribution including the fixed pattern noise is improved. Is greatly reduced. Therefore, an alarm /
An error occurs in the control. For example, in an image processing apparatus that detects a white object such as lane marking, false white pixels of fixed pattern noise generated at high temperatures may be erroneously detected and erroneous lane recognition may be performed. If a lane departure warning or vehicle control is performed during this time, a dangerous situation may occur. Because of the problem of such fixed pattern noise, Japanese Patent Application Laid-Open
As in the case of the camera with a temperature detecting device disclosed in Japanese Patent Application Laid-Open No. 0-347284, it is necessary to consider not only a problem relating to mechanical parts but also an image sensor.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and has as its object to provide an image processing apparatus which can reliably recognize an object in real time.

[0005]

In an image processing apparatus according to the present invention, an image pickup means for picking up an image, a recognition image processing means for extracting a target from an image picked up by the image pickup means, Temperature measuring means for measuring the temperature, and a communication means for transmitting the output of the recognition image processing means, the communication means, when the temperature measured by the temperature measuring means is not within a predetermined range, a predetermined signal What to send. Further, in an image processing apparatus mounted on a vehicle and constituting a warning system together with a control unit for giving a warning about vehicle running, an image pickup means for picking up an image, and a recognition image for extracting an object from the image picked up by the image pickup means Processing means, temperature measuring means for measuring the temperature of the imaging means, and communication means for transmitting the output of the recognition image processing means to the control unit, wherein the communication means has a temperature measured by the temperature measuring means within a predetermined range. When not in
A predetermined signal is transmitted to the control unit.

[0006] The predetermined signal is a signal for requesting temperature adjustment of the imaging means. Further, the predetermined signal is a recognition result use prohibition signal for prohibiting use of the output of the recognition image processing means.

Further, even when the temperature of the imaging means measured by the temperature measurement means is not within a predetermined range, the recognition image processing means continuously extracts the object.
The image processing apparatus further includes a recognition reliability assurance temperature setting unit that sets a temperature that guarantees the reliability of recognition by the recognition image processing unit.

[0008]

Embodiments of the present invention will be described below with reference to the drawings. Embodiment 1 FIG. FIG. 1 is a simplified configuration diagram showing an image processing apparatus according to Embodiment 1 of the present invention. In FIG. 1, reference numeral 1 denotes a photographing means, 2 denotes an image storage means for temporarily storing the whole or a part of an image obtained by the photographing means 1, 3
Is a recognition image processing means for reading out image data from the image storage means 2 and extracting an object in real time; 4, a temperature measuring means for measuring the temperature of the photographing means 1; and 5, a recognition image processing result by the recognition image processing means 3. And communication means for receiving and transmitting the result of the temperature measurement by the temperature measurement means 4 to the outside. FIG. 2 is a flowchart showing a processing flow of the image processing apparatus according to the first embodiment of the present invention.

Next, the operation will be described with reference to the flowchart of FIG. In step S1, the object is photographed by the photographing means 1, and in step S2, the photographed image is fetched into the image memory of the image storage means 2 as luminance value data and temporarily stored. Note that the luminance value data to be stored may be only an area related to object recognition. Next, in step S3, the recognition image processing means 3 reads the luminance value data from the image memory, and extracts the object based on the time-series and spatial characteristics of the luminance distribution.

In step S4, the temperature of the imaging means 1 is acquired by the temperature sensor of the temperature measuring means 4, and it is determined whether or not the temperature is a temperature at which fixed pattern noise is generated, that is, whether or not the temperature is within the temperature range for guaranteeing the recognition reliability. The temperature measurement is desirably performed on the image sensor, but if it is difficult due to the camera mounting conditions, the temperature around the camera may be used instead. In step S4, when the temperature measuring means 4 determines that the temperature is within the recognition reliability guarantee temperature range, in step S5,
The image recognition result is output to the control unit by the communication unit 5. In step S4, if the temperature measuring means 4 determines that the temperature is out of the recognition reliability assurance temperature range, the communication means 5 outputs a recognition result use prohibition signal and a temperature adjustment request (predetermined signal) to the control unit in step S6. I do. In this case, a flag may be prepared for normal output data, or the recognition result data may be reset to 0 instead of the flag. The control unit that receives the message displays a message urging the user to adjust the temperature or automatically performs the temperature adjustment.

According to the first embodiment, the occurrence of fixed pattern noise in the photographing means is determined by the temperature measuring means for the photographing means, and the recognition result for an image having an abnormal image quality is prohibited. Alarm
It is possible to avoid erroneous control alarm and control as a system. Further, the provision of the temperature measurement means for the imaging means enables the temperature measurement without increasing the processing time for recognition.

Embodiment 2 FIG. The image processing apparatus according to the first embodiment will be described using a lane recognition image processing apparatus used for a lane keeping and a lane departure warning system (warning system) as an example. FIG. 3 is an overall configuration diagram in which the image processing apparatus according to Embodiment 2 of the present invention is applied to a vehicle-mounted camera. In FIG. 3, reference numeral 6 denotes an image processing unit, which has an image sensor 7 serving as a photographing unit and a temperature sensor 8 serving as existing temperature measuring means for the image sensor 7. 9 is a control unit connected to the image processing unit 6, 10 is an actuator controlled by the control unit 9, 11 is an alarm buzzer controlled by the control unit 9, 12 is a defroster controlled by the control unit 9, 6 ~ 1
2 configures the system.

FIG. 4 is a view showing a normal vehicle-mounted camera output image using the image processing apparatus according to the second embodiment of the present invention. In FIG. 4, reference numeral 13 denotes an output image of the vehicle-mounted camera taken in front of the vehicle, and reference numeral 14 denotes lane markings of the own lane, all of which are within the angle of view. FIG.
FIG. 9 is a diagram showing an output image of a vehicle-mounted camera including fixed pattern noise at high temperature using the image processing apparatus according to the second embodiment of the present invention. In FIG. 5, reference numerals 13 and 14 are the same as those in FIG. Reference numeral 15 denotes a fixed pattern noise generated at a high temperature in the output image 13 of the vehicle-mounted camera which photographs the front of the vehicle.

FIG. 6 is a diagram showing the result of lane marking recognition in the image processing of FIG. 4 and showing normal recognition.
5 shows a lane recognition result for the output image 13 of the camera shown in FIG. 6, 13 and 14 are the same as those in FIG. Reference numeral 16 denotes a recognition result of the left and right lane markings 14, which are represented by thick lines. In this example, the inside of the lane marking is set as the detection position, and the lane marking is approximated by a quadratic expression. FIG. 7 is a diagram illustrating the lane marking recognition result of the image processing in FIG. 5, which is an explanatory diagram illustrating erroneous recognition, and illustrates the lane recognition result with respect to the output image 13 of the camera illustrated in FIG. 5. In FIG.
15 is the same as that in FIG. 16 is a recognition result of the lane marking 14. Fixed pattern noise 15 of the same brightness level near lane marking 14
Has occurred, a part of the fixed pattern noise 15 is erroneously detected as the position of the lane marking 14, and the recognition result 1
6 is significantly different from the actual lane marking position 14.

When the control unit 9 instructs the actuator 10 to keep the lane or issues a lane departure warning to the warning buzzer 11 based on the recognition result shown in FIG. 7, the control becomes different from the actual lane shape. , Could lead to dangerous situations. It is determined from the temperature of the image sensor 7 obtained by the temperature sensor 8 that the output image has a possibility of erroneous recognition as shown in FIG. A result use prohibition signal and a temperature adjustment request are output. The control unit 9 that has received the recognition result use prohibition signal does not issue a command for performing lane keeping or a lane departure warning to the actuator 10 and the warning buzzer 11. When the image processing unit 6 is mounted in the vehicle interior, for example, in the rear of the rearview mirror, the control unit 9 automatically operates the defroster 12 of the car air conditioner in order to lower the temperature around the image processing unit 6, or Prompt the driver for operation. When the temperature of the image sensor 7 falls within the allowable range due to an external environment such as cooling means for the image processing unit 6 or a decrease in outside air temperature, the image processing unit 6 determines this by the temperature sensor 8 and automatically determines the temperature. Release the recognition result use prohibition signal.

According to the second embodiment, in the lane recognition image processing apparatus, when a fixed pattern noise is generated in the imaging means, the existing temperature adjustment means is applied, so that the temperature adjustment means is not added to the image processing apparatus. It is possible to avoid false alarm / error control of alarm / control based on the recognition result as a system.

Embodiment 3 The third embodiment relates to a case where the temperature of the imaging unit returns from outside the recognition reliability temperature range in the image processing apparatus according to the second embodiment. If the temperature of the imaging unit is outside the recognition reliability temperature range, the output of the recognition image processing unit is masked by outputting the recognition result use prohibition signal without stopping the image processing, and the temperature is within the recognition reliability temperature range. Minimize output interruption time when returning to. The processing flow of the third embodiment is the same as that shown in FIG.
After performing step 3, check the temperature in step S4,
It is determined whether the output is based on step S5 or step S6.

FIG. 8 is an explanatory diagram showing the left and right lane marking search areas immediately after the start of lane marking image processing using the image processing apparatus according to the third embodiment of the present invention or in a state where lane markings have not been extracted. In FIG. 8, 1
3, 14, and 16 are the same as those in FIG. Reference numerals 17 and 18 denote left and right lane marking search areas immediately after the start of lane marking image processing or in a state in which lane marking has not been extracted. Reference numeral 17 denotes a left lane marking search area, and reference numeral 18 denotes a right lane marking search area. FIG.
FIG. 9 is an explanatory diagram showing left and right lane marking tracking areas when lane markings are extracted using the image processing device according to the third embodiment of the present invention. In FIG. 9, 13, 14,
16 is the same as that in FIG. 19, 2
0 is a left and right lane marking tracking area at the time of lane marking extraction, 19 is a left lane marking tracking area,
Reference numeral 20 denotes a right lane marking tracking area.

Hereinafter, it will be described that restarting from the image processing stopped state takes time until the recognition is restored. FIG.
Since there is no prior information on where the lane marking is located on the screen so as to indicate the left and right lane marking search areas 17 and 18 immediately after the lane marking image processing is started or the lane marking is not extracted, it is necessary to search from a wide area. And it takes a long processing time to extract.
On the other hand, since the tracking area can be set based on the previous extraction position as shown by the left and right lane marking tracking areas 19 and 20 at the time of lane marking extraction in FIG. Tracking can be limited to a small area. Once the image processing is stopped, the image processing must be restarted from the state shown in FIG. 8 and it takes a long time to recover the recognition. However, if the image processing is not stopped and the recognition result output mask is used, the processing shown in FIG. The image processing can be restarted from the state described above, and the recognition and recovery are performed immediately.

According to the third embodiment, when the temperature of the imaging means returns from outside the allowable temperature range to within the allowable range, the image processing is not stopped, so that the recognition result can be output immediately. It is possible to minimize the interruption time of use.

Embodiment 4 FIG. In the fourth embodiment, the image processing apparatus according to the second embodiment is provided with a recognition reliability assurance temperature setting means for setting a recognition reliability assurance temperature for each image processing apparatus. FIG. 10 is a diagram showing an image in which fixed pattern noise is generated outside the left and right lane marking search areas using the image processing apparatus according to Embodiment 4 of the present invention. 10, 13, 14 and 15 are the same as those in FIG. 5, and 17 and 18 are the same as those in FIG. FIG. 11 is a diagram showing an image in which fixed pattern noise is generated in the left and right lane marking search areas using the image processing device according to the fourth embodiment of the present invention, which may cause erroneous recognition. In FIG. 11, 1
3, 14, 15, 17, 18 are the same as those in FIG.

The occurrence of fixed pattern noise is unique to each image sensor, and has the characteristic that pixels at positions determined according to the temperature region are overexposed and underexposed. For example, at a certain temperature, as shown in FIG.
Is assumed to have occurred. At this time, since the fixed pattern noise is not included in the areas to be referred to in the trapezoidal image processing, that is, the left and right lane marking search areas 17 and 18, the recognition reliability is not affected. Since the left and right lane marking tracking areas 19 and 20 in FIG. 9 are included in the left and right lane marking search areas 17 and 18, only the left and right lane marking search areas 17 and 18 will be described here.
When the temperature further rises from the state shown in FIG. 10, fixed pattern noise 15 is generated in the left and right lane marking search areas 17 and 18 as shown in FIG. That is, the left and right lane marking search area 17,
The temperature at which the fixed pattern noise 15 is generated within 18 is the limit temperature for guaranteeing the recognition reliability of the image processing apparatus. In the fourth embodiment, since the limit temperature of the recognition reliability guarantee is unique to each image processing apparatus, the recognition reliability guarantee temperature is set for each image processing apparatus.

According to the fourth embodiment, it is possible to set the recognition reliability assurance temperature beyond the fixed pattern noise occurrence temperature of the image pickup device while securing the further reliability of the recognition reliability assurance temperature.

[0024]

Since the present invention is configured as described above, it has the following effects. Image capturing means for capturing an image, recognition image processing means for extracting an object from the image captured by the image capturing means, temperature measuring means for measuring the temperature of the image capturing means, and communication for transmitting the output of the recognition image processing means The communication means transmits a predetermined signal when the temperature measured by the temperature measurement means is not within a predetermined range. By using the result of the above, it is possible to avoid performing an erroneous alarm / control.

Further, in an image processing apparatus mounted on a vehicle and constituting an alarm system together with a control unit for giving an alarm about the running of the vehicle, an image pickup means for picking up an image, and an object is extracted from the image picked up by the image pickup means. And a communication unit for transmitting the output of the recognition image processing unit to the control unit, wherein the communication unit measures the temperature measured by the temperature measurement unit. When it is not within the range, the predetermined signal is transmitted to the control unit, so that the control unit uses the result of erroneous recognition of the recognition image processing apparatus due to the occurrence of fixed pattern noise in the photographing means to perform erroneous alarm / control. Can be avoided.

Further, since the predetermined signal is a signal requesting the temperature adjustment of the image pickup means, the temperature of the image pickup means can be adjusted. Further, since the predetermined signal is a recognition result use prohibition signal for prohibiting use of the output of the recognition image processing means, use of the output of the recognition image processing means can be prohibited.

Further, even when the temperature of the image pickup means measured by the temperature measuring means is not within the predetermined range, the recognition image processing means continuously extracts the object, so that the output of the recognition image processing means is output. Use can be resumed quickly.
In addition, since the recognition reliability assurance temperature setting means for setting the temperature that guarantees the reliability of the recognition by the recognition image processing means is provided for the temperature of the imaging means, the recognition reliability assurance temperature setting is performed for each image processing apparatus. Can be.

[Brief description of the drawings]

FIG. 1 is a simplified configuration diagram showing an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a processing flow of the image processing apparatus according to the first embodiment of the present invention.

FIG. 3 is an overall configuration diagram in which an image processing apparatus according to a second embodiment of the present invention is applied to an in-vehicle camera.

FIG. 4 is a diagram showing a normal vehicle-mounted camera output image using the image processing device according to the second embodiment of the present invention;

FIG. 5 is a diagram showing an output image of a vehicle-mounted camera including fixed pattern noise at a high temperature using the image processing apparatus according to the second embodiment of the present invention;

6 is an explanatory diagram showing lane marking recognition results of the image processing of FIG. 4 and showing normal recognition.

FIG. 7 is an explanatory diagram showing lane marking recognition results of the image processing in FIG. 5 and showing erroneous recognition.

FIG. 8 is an explanatory diagram illustrating a left and right lane marking search area immediately after the start of lane marking image processing using the image processing apparatus according to the third embodiment of the present invention or in a state in which lane marking has not been extracted.

FIG. 9 is an explanatory diagram illustrating left and right lane marking tracking areas at the time of lane marking extraction using the image processing device according to the third embodiment of the present invention.

FIG. 10 is a diagram showing an image in which fixed pattern noise is generated outside a left and right lane marking search area using an image processing apparatus according to a fourth embodiment of the present invention.

FIG. 11 is a diagram showing an image in which fixed pattern noise is generated in a left and right lane marking search area using the image processing apparatus according to Embodiment 4 of the present invention, which may cause erroneous recognition.

[Explanation of symbols]

REFERENCE SIGNS LIST 1 shooting means, 2 image storage means, 3 recognized image processing means, 4 temperature measurement means, 5 communication means, 6 image processing unit, 7 image sensor, 8 temperature sensor, 9 control unit, 10 actuator, 11 alarm buzzer, 12
Defroster, 13 camera output image, 14 lane marking, 15 fixed pattern noise, 16 lane marking recognition result, 17 left lane marking search area, 18 right lane marking search area, 19 left lane marking tracking area, 20 right lane marking tracking area.

Claims (6)

[Claims] An imaging unit that captures an image; a recognition image processing unit that extracts an object from an image captured by the imaging unit; a temperature measurement unit that measures a temperature of the imaging unit;
Communication means for transmitting an output of the recognition image processing means, wherein the communication means transmits a predetermined signal when the temperature measured by the temperature measurement means is not within a predetermined range. Image processing device. 2. An image processing apparatus which is mounted on a vehicle and constitutes an alarm system together with a control unit which issues an alarm about running of the vehicle. An image pickup means for picking up an image, and an object is extracted from the image picked up by the image pickup means. Recognition image processing means, temperature measurement means for measuring the temperature of the imaging means, communication means for transmitting the output of the recognition image processing means to the control unit, the communication means, the temperature measured by the temperature measurement means Transmitting a predetermined signal to the control unit when is not within a predetermined range. 3. The image processing apparatus according to claim 1, wherein the predetermined signal is a signal requesting a temperature adjustment of the imaging unit. 4. The signal according to claim 1, wherein the predetermined signal is a recognition result use prohibition signal for prohibiting use of an output of the recognition image processing means. Image processing device. 5. The apparatus according to claim 1, wherein the recognition image processing means continuously extracts the object even when the temperature of the imaging means measured by the temperature measurement means is not within a predetermined range. The image processing apparatus according to claim 1. 6. The apparatus according to claim 1, further comprising a recognition reliability assurance temperature setting means for setting a temperature at which the reliability of recognition by said recognition image processing means is guaranteed for the temperature of said imaging means. 6. The image processing device according to claim 5.