US20170041591A1

US20170041591A1 - Vehicle-Mounted Image Recognition Device

Info

Publication number: US20170041591A1
Application number: US15/102,282
Authority: US
Inventors: Naoki KOROGI; Yuji Otsuka
Original assignee: Hitachi Automotive Systems Ltd
Current assignee: Hitachi Astemo Ltd
Priority date: 2013-12-25
Filing date: 2014-11-12
Publication date: 2017-02-09
Also published as: WO2015098325A1; EP3089442A1; JP6325000B2; JPWO2015098325A1; EP3089442B1; EP3089442A4

Abstract

This vehicle-mounted image recognition device achieves improved performance in flashing light source recognition at low cost and without loss of performance in image recognition applications such as preceding vehicle recognition and white line recognition. At staggered exposure timing, imaging is performed alternately with a first imaging unit and a second imaging unit, which form a pair in a vehicle-mounted camera. A flashing light source is detected from one or both of the first image obtained from the first imaging unit and a second image obtained from the second imaging unit, and the following times, exposure of the first imaging unit and the second imaging unit is matched to when the flashing light source is on. Thereby, flashing light sources are recognized with high accuracy without loss of performance of the image recognition function.

Description

TECHNICAL FIELD

The present invention relates to an in-vehicle camera, and particularly to an in-vehicle image recognition device having a function of simultaneously recognizing a plurality of targets such as a preceding vehicle, a white lane, a pedestrian, and a light source.

BACKGROUND ART

In recent years, light sources used in a traffic signal and a traffic sign of an electric light display type are replaced from the conventional incandescent lamps to LEDs. The LED has a merit on less power consumption and long life span compared to the incandescent lamp. In addition, there is no pseudo lighting considered as emission caused by the reflection of sunlight, so that the visibility is high and also the stability is improved.
The LED used in the traffic signal and the traffic sign is flickered to be repeatedly on and off at a high speed when being turned on. A cycle of the flickering is 100 Hz in a region of 50 Hz commercial power source or 120 Hz in a region of 60 Hz commercial power source. The flickering is performed at a high speed, and thus it appears to person as always turned on.
In a case where the exposure timing of an in-vehicle camera is matched to the OFF state of the flickering light source, the emission light of the light source is not recorded in the image obtained from the camera. In addition, in a case where the capturing cycle of the camera is close to an integer times the cycle of the flickering light source, the above problem continuously occurs in a plurality of frames.
In order to solve the above problem, for example, PTL 1 discloses a technique in which a number of imaging units are provided to capture an image while deviating the exposure timings in a time shorter than a flickering cycle of the flickering light source and to observe a change in brightness of the light source in order to recognize the flickering light source.

CITATION LIST

Patent Literature

PTL 1: Japanese Patent Application Laid-Open No. 2007-286943

SUMMARY OF INVENTION

Technical Problem

However, in the conventional technique described above, an image is acquired in a cycle Tled shorter than the flickering cycle of the light source, and a recognition process is performed on each image, thereby significantly increasing a processing load. In addition, since a number of images are transferred from an imaging unit to a recognition unit, a transfer amount of a bus may be significantly expanded. An in-vehicle camera having an image recognition function is requested for simultaneously recognizing a number of objects such as a preceding vehicle, a pedestrian, and a white lane on a road. Therefore, when the processing load and the bus transfer amount are increased, it becomes difficult to be compatible with the other recognition processing in addition, since more imaging units are required, the cost is increased.
The invention has been made in view of the above problems, and an object thereof is to provide an in-vehicle imaging device which can improve a performance of recognizing the flickering light source at a low cost without causing a degradation of an image recognition application such as a preceding vehicle recognition and a white lane recognition.

Solution to Problem

There is provided an in-vehicle image recognition device according to the present invention in order to solve the above issue which processes an image captured by a pair of imaging units, wherein the same light source is captured by the pair of imaging units in a capturing cycle which is an integer times a predetermined flickering cycle, and a capturing cycle where exposure timings of the pair of imaging units are deviated by a half of the flickering cycle, and wherein, in a case where a difference between a pixel value of the light source captured by one imaging unit and a pixel value of the light source captured by the other imaging unit is equal to or more than a threshold, the light source is determined as a flickering light source.

Advantageous Effects of Invention

According to the invention, the visibility of a flickering light source can be improved at a low cost without causing a degradation of the performance of an image recognition application such as a preceding vehicle recognition and a white lane recognition. Further, other objects, configurations, and effects will become clear through the explanation of the following embodiments.

BRIEF DESCRIPTION OF DRAWINGS

[FIG. 1] FIG. 1 is a block diagram for describing functions of an in-vehicle image recognition device.

[FIG. 2] FIG. 2 is a diagram illustrating an example of flickering of a flickering light source and exposure timings of a first imaging unit and a second imaging unit.

[FIG. 3] FIG. 3 is a flowchart illustrating an exemplary operation of an exposure timing adjustment unit.

DESCRIPTION OF EMBODIMENTS

Next, embodiments of the invention will be described below using the drawings. FIG. 1 is a functional block diagram illustrating an example of an in-vehicle image recognition device in this embodiment.
An in-vehicle image recognition device 10 is a device which alternately captures an image at a different timing using a first imaging unit 11 a and a second imaging unit 11 b which are paired in an in-vehicle camera, detects a flickering light source from one or both of a first image 12 a obtained from the first imaging unit 11 a and a second image 12 b obtained from the second imaging unit 11 b, and performs exposure using the first imaging unit 11 a and the second imaging unit 11 b in synchronization with timing of turning on the flickering light source from next time, so that the flickering light source is recognized with high accuracy without causing degradation of the performance of an image recognition function.
The in-vehicle image recognition device 10 is a stereo camera which is mounted in a vehicle and, as described in the drawing, provided with the first imaging unit 11 a and the second imaging unit 11 b, a distance data generation unit 13 which generates distance information based on the first image 12 a and the second image 12 b acquired by the first imaging unit 11 a and the second imaging unit 11 b, a distance data storage unit 14 which stores the distance information indicating a distance to a capturing target, an image recognition unit 15 which performs image recognition, a light source recognition unit 16 which performs recognition of a light source, an exposure timing adjustment unit 17 which adjusts each timing of exposure, an external information reception unit 18 which receives external information, and a first exposure control unit 19 a and a second exposure control unit 19 b which control the exposure of the first imaging unit 11 a and the second imaging unit 11 b.
The first imaging unit 11 a and the second imaging unit 11 b are directed in the same direction and capture the same range. For example, the imaging units are separately disposed on right and left in the vicinity of a rear-view mirror of the vehicle to capture the forward side of the vehicle through a front class
The distance data generation unit 13 generates the distance information such as a parallax image stereoscopically obtained by the first image 12 a and the second image 12 b. The image recognition unit 15 recognizes an object body such as a traffic signal, a traffic sign, a preceding vehicle, a pedestrian, and a bicycle using at least one of the first image 12 a and the second image 12 b. As a recognition method, the well-known pattern matching may be used.
The light source recognition unit 16 recognizes a light source from at least one of the first image 12 a and the second image 12 b, and recognizes whether the recognized light source is a flickering light source. The exposure timing adjustment unit 17 adjusts the exposure timings of the first imaging unit 11 a and the second imaging unit 11 b based on an image recognition result of the image recognition unit 15 and the external information received by the external information reception unit 18. The first exposure control unit 19 a and the second exposure control unit 19 b control the exposure timings of the first imaging unit 11 a and the second imaging unit 11 b according to, for example, an instruction from the exposure timing adjustment unit 17. In addition, the exposure timings are adjusted to be exposure parameters optimized to the image recognition based on the information of the images acquired from the first imaging unit 11 a and the second imaging unit 11 b. The external information reception unit 18 is connected to a vehicle network 20 and acquires the external information such as vehicle speed information 21, a yaw rate 22, GPS information 23, time information 24, and map data 25 using the external information reception unit 18.
When capturing the traffic signal using an LED as the flickering light source and an indicator such as the traffic sign of an electric light display type, the vehicle image recognition device 10 captures the images at exposure timings of the first imaging unit 11 a and the second imaging unit 11 b in synchronization with a flickering period of the LED.
Specifically, capturing cycles Tc of the first imaging unit 11 a and the second imaging unit 11 b are set to be Nc times (an integer) of a flickering cycle Tl, and the exposure timing of the first imaging unit 11 a and the exposure timing of the second imaging unit 11 b are set to be deviated by Ns (an integer)+0.5 times the flickering cycle, so that the first image and the second image deviated by a half cycle with respect to the flickering cycle are acquired.
Therefore, the light source in the ON state is recorded in at least any one of the first image 12 a and the second image 12 b.
The light source recognition unit 16 performs a recognition process in which the light source is extracted from the first image 12 a and the second image 12 b. Then, in a case where there is a difference equal to or more than a predetermined value between pixel values of the light source extracted from the first image 12 a and the second image 12 b, it is determined that the subject light source is a flickering light source. In a case where the light source recognition unit 16 determines that the light source is a flickering light source, the exposure timing adjustment unit 17 changes the exposure timing such that the exposure timing of one imaging unit is changed to be matched to the exposure timing of the other imaging unit in which the flickering light source is brightly captured among the first imaging unit 11 a and the second imaging unit 11 b, and adjusts the exposure timings of the first imaging unit 11 a and the second imaging unit 11 b to be synchronized. The distance data generation unit generates distance data indicating a distance from the vehicle to the light source based on the first image and the second image captured by the first imaging unit and the second imaging unit.
Next, an example of adjusting the exposure timing of the invention will be described using FIG. 2. FIG. 2 is a diagram illustrating an example of flickering of the flickering light source and exposure timings of the first imaging unit and the second imaging unit. As illustrated in FIG. 2, in a case where the brightness of the LED is increased or decreased in the flickering cycle Tl and the flickering light source emits the light, the exposure timing adjustment unit 17 sets the capturing cycles Tc of the first imaging unit 11 a and the second imaging unit 11 b to be Nc (an integer) times the flickering cycle Tl. Therefore, even the flickering light source can be captured in the same brightness in every frame, and the brightness of the flickering light source illuminated on each frame can be made constant.
Then, the exposure timing adjustment unit 17 sets a deviation time between the exposure timing of the first imaging unit and the exposure timing of the second imaging unit to be Ns (an integer)+0.5 times the flickering cycle Tl (Deviation Time=Tl×(Ns+0.5)). Therefore, the first imaging unit 11 a and the second imaging unit 11 b can acquire brightnesses deviated by the half flickering cycle Tl of the LED. Herein, in a case where the exposure timing of one of the first imaging unit 11 a and the second imaging unit 11 b is a timing when the flickering light source is dark, the flickering light source at the exposure timing of the other imaging unit necessarily comes to be bright since the exposure timing of the other imaging unit is deviated by the half cycle with respect to the flickering cycle Tl.
The light source recognition unit 16 can recognize that the light source is flickering by recognizing that there is a light source in the image obtained by an imaging unit and there is no light source in the image obtained by the other imaging unit. Therefore, it is possible to reliably capture the flickering light source from next time by causing the first imaging unit 11 a and the second imaging unit 11 b to perform exposure in synchronization with timing when the flickering light source is bright. Therefore, the flickering light source can be stereoscopically viewed, and accurate distance information up to the flickering light source can be obtained.
Even at a timing when the flickering light source is captured in the same brightness by the first imaging unit 11 a and the second imaging unit 11 b, the traffic signal and the indicator such as the traffic sign using the LED are sufficiently bright compared to the preceding vehicle and a road surface in order to secure visibility. Therefore, even in a case where an optimal exposure is performed in order to recognize the preceding vehicle and the road surface, the flickering light source can be captured with a sufficient pixel value, and can be recognized similarly to an unflickering light source (a light source always turned on).
In this case, the light source can be recognized in synchronization with the exposure timing of any one of the first imaging unit 11 a and the second imaging unit 11 b. Therefore, the flickering light source can be recognized and the distance information can be obtained by performing the stereoscopic viewing in synchronization with the exposure timing of any one of the imaging units. In other words, the flickering light source can be recognized similarly to a case where the light always turned on is recognized.
Further, as a method of recognizing the light source from an image, any method such as a template matching and a classification using color information may be used as long as a feature can be extracted from the image, and a well-known method may be used.
FIG. 3 is a flowchart for describing an exemplary method of setting the exposure timing using the exposure timing adjustment unit. The exposure timing adjustment unit 17 determines whether the stereoscopic distance data is generated based on the recognition result of the image recognition unit 15 and the information acquired from the external information reception unit 18.
First, in Step S1, recognition result information is received from the image recognition unit 15 and the light source recognition unit 16. Next, in Step S2, the external information such as the vehicle speed information 21, the yaw rate 22, the GPS information 23, the time information 24, and the map data 25 linked to the vehicle network 20 are acquired from the external information reception unit 18.
Then, in Step S3, it is determined whether the flickering cycle Tl of the flickering light source is 1/50 second. Since the capturing cycle Tc and a capture deviation time are determined based on the flickering cycle of the flickering light source, the flickering cycle of the flickering light source is necessarily determined.
In Step S3, in a case where it is determined that the flickering cycle of the flickering light source is 1/50 second, the flickering cycle of the flickering light source is set to 1/50 second in Step S4, and if not (that is, it is determined that the flickering cycle is not 1/50 second), the flickering cycle of the flickering power source is set to 1/60 second in Step S5.
At the first time, since the flickering cycle is not clear, the flickering cycle is set to any one of 1/50 and 1/60 second. For example, in a case where the pixel value of the light source is almost the same value in the image captured by the same imaging unit, it can be determined that the capturing cycle is an integer times an intended flickering cycle and the flickering cycle is correctly set.
On the other hand, in a case where the pixel value of the light source is changed by stages in the image captured by the same imaging unit, it is determined that the flickering cycle of the light source is different from the current setting, and thus the setting of the flickering cycle is switched to the other value. The flickering cycle from next time is set to the cycle at the previous time. In a case where it is determined that the flickering cycle is different on the same basis, the setting is switched.
The flickering cycle of the flickering light source may be determined based on a power source frequency of the flickering light source. Since the frequency of the commercial power source is determined depending on regions, the flickering cycle may be determined according to GPS information or a running location of a subject vehicle received in cooperation with a car navigation system. For example, in the case of Japan, the power source frequency in the east region of Japan is 50 Hz, and the power source frequency in the west region of Japan is 60 Hz. Therefore, the power source frequency of the flickering power source can be recognized by specifying the current location of the vehicle based on the external information such as the GPS information 23 and the map data 25. For example, in a case where the power source frequency is 50 Hz, and in a case where the flickering cycle of the flickering light source is 1/50 second and the power source frequency is 60 Hz, the flickering cycle of the flickering light source can be set to 1/60 second.
Next, in Step S6, it is determined whether a light emission timing of the flickering light source is estimated. In a case where the light emission timing of the flickering light source is estimated, the first imaging unit 11 a and the second imaging unit 11 b are caused to simultaneously perform the capturing in synchronization with the light emission timing of the flickering light source in Step S8. In this case, the flickering light source can reliably perform the capturing using both the first imaging unit 11 a and the second imaging unit 11 b. Further, the distance information up to the light source can be stereoscopically acquired.
On the other hand, in a case where the light emission timing of the flickering light source is not estimated yet, it is determined whether the distance information is necessary in Step S7. While the acquirement of the distance information using the stereo camera is a feature, the distance information may be not necessary for all the frames. Therefore, the stereoscopic viewing (the imaging units are caused to perform the capturing at the same exposure timing) or the deviating of the exposure timings is selected according to a situation based on the recognition result acquired in Step S1 and the external information acquired in Step S2. In a case where it is determined that the distance information is necessary, the first imaging unit and the second imaging unit are caused to simultaneously perform the capturing in synchronization with the light emission timing of the flickering light source in Step S8. In a case where it is determined that the distance information is not necessary, the capturing is performed by deviating the exposure timings of the first imaging unit and the second imaging unit by a predetermined time in Step S9.
For example, in a case where it is a situation that the distance information from the stereo camera is necessary for the other applications such as a pedestrian detection and a preceding vehicle detection, the capturing is performed at the same exposure timing based on the other applications with priority higher than that of the light emission timing of the flickering light source.
In the in-vehicle image recognition device 10, the preceding vehicle and the pedestrian are recognized from the image, and the result is acquired from the image recognition unit 15. In a case where the preceding vehicle is within a predetermined distance, or in a case where the pedestrian or the bicycle is near the advancing path of the subject vehicle, it is determined that the distance information is necessary for avoiding a risk. In addition, a three-dimensional object can be effectively detected in the stereoscope viewing. Therefore, a counter is provided in the device, and the capturing may be performed alternately using the stereoscopic viewing and the deviating of the exposure timings in a predetermined cycle even in a case where there is no change in an ambient environment.
In a case where the traffic signal or the traffic sign of an electric light display type can be acquired from the map data 25, the exposure timings may be deviated only when the subject vehicle comes close to a predetermined distance or more. It is desirable that a parameter related to the exposure be optimal to recognize the preceding vehicle and the pedestrian. However, for example, in a case where it is determined that the light source is recognized with priority, a shutter opening time may be lengthened.
According to the in-vehicle image recognition device 10 described above, the shutter opening time and a pixel transfer amount of the imaging unit are not changed, so that the performance of the image recognition application is not damaged. In addition, since the extraction process of the light source is performed as in the conventional process, a processing load is not increased. Further, there is no increase in hardware, the recognition performance of the flickering light source can be improved at a low cost.
Hitherto, the description has been made about embodiments of the invention, but the invention is not limited to the embodiments. Various changes in design can be made within a scope not departing from the spirit of the invention described in the accompanying claims. For example, the embodiments are described in a clearly understandable way for the invention, and thus the invention is not necessarily to include all the configurations described above. In addition, some configurations of a certain embodiment may be replaced with the configurations of another embodiment, and the configuration of the other embodiment may also be added to that of a certain embodiment. Furthermore, various additions, omissions, and substitutions may be partially made in some of the configurations of the respective embodiments.

REFERENCE SIGNS LIST

10 in-vehicle image recognition device
11 a first imaging unit
11 b second imaging unit
12 a first image
12 b second image
13 distance data generation unit
14 distance data
15 image recognition unit
16 light source recognition unit
17 exposure timing adjustment unit
18 external information reception unit
19 a first exposure control unit
19 b second exposure control unit
20 vehicle network
21 vehicle speed information
22 yaw rate
23 GPS information
24 time information
25 map data

Claims

1. An in-vehicle image recognition device which processes an image captured by a pair of imaging units,

wherein the same light source is captured by the pair of imaging units in a capturing cycle which is an integer times a predetermined flickering cycle, and a capturing cycle where exposure timings of the pair of imaging units are deviated by a half of the flickering cycle, and

wherein, in a case where a difference between a pixel value of the light source captured by one imaging unit and a pixel value of the light source captured by the other imaging unit is equal to or more than a threshold, the light source is determined as a flickering light source.

2. The in-vehicle image recognition device according to claim 1,

wherein, in a case where the light source is determined as the flickering light source, the exposure timing of the imaging unit is matched to the exposure timing of the other imaging unit of which the captured flickering light source is bright in the pair of imaging units, and distance data up to the light source is generated based on a pair of images captured by the pair of imaging units at the same exposure timing.

3. The in-vehicle image recognition device according to claim 2,

wherein two types of flickering cycles are set in advance,

wherein, in a case where the pixel value of the light source is constant in the image captured by the same imaging unit, it is determined that the setting of the flickering cycle is correct, and

wherein, in a case where the pixel value of the light source is changed by stages, it is determined that the setting of the flickering cycle is wrong, and the setting of the flickering cycle is switched to the other setting.

4. The in-vehicle image recognition device according to claim 1, comprising:

a first imaging unit;

a second imaging unit which is disposed in parallel to the first imaging unit;

a distance data generation unit which generates distance data based on images acquired from the first imaging unit and the second imaging unit; and

a light source recognition unit which recognizes the light source based on the images acquired from the first imaging unit and the second imaging unit,

wherein the first imaging unit and the second imaging unit perform capturing in a capturing cycle based on a flickering cycle of the recognized flickering light source and in a capturing cycle obtained by deviating exposure timings of the first imaging unit and the second imaging unit by a half of the flickering cycle, and

wherein the light source recognition unit determines whether the light source is a flickering light source based on an image captured by the first imaging unit and an image captured by the second imaging unit.

5. The in-vehicle image recognition device according to claim 4, comprising:

an image recognition unit which performs image recognition based on the images acquired from the first imaging unit and the second imaging unit and the distance data;

an external information reception unit which receives information containing at least one of a vehicle speed, a yaw rate, GPS information, time information, and map data; and

an exposure timing adjustment unit which adjusts the exposure timing based on a recognition result of the light source recognition unit, a recognition result of the image recognition unit, and the information of the external information reception unit.

6. The in-vehicle image recognition device according to claim 5, comprising:

a first exposure control unit and a second exposure adjustment unit which adjust the exposure timing to be an exposure parameter optimized to the image recognition based on the information of the images acquired by the first imaging unit and the second imaging unit.

7. The in-vehicle image recognition device according to claim 5,

wherein the exposure timing adjustment unit estimates the flickering cycle of the flickering light source and adjusts the exposure timing based on the recognition result of the light source recognition unit.

8. The in-vehicle image recognition device according to claim 5,

wherein the exposure timing adjustment unit determines whether the distance data is stereoscopically generated based on the recognition result of the image recognition unit and the information acquired from the external information reception unit.