KR101606476B1 - Apparatus and method for recognition signal light using multiple exposure image of camera - Google Patents

Abstract

The apparatus for automatically recognizing a traffic light according to an embodiment of the present invention includes a photographing unit for photographing the front of a vehicle, a multiple exposure image capturing unit for capturing multiple exposure images continuously photographed through the photographing unit according to the exposure time, A color of the signal light detected by the color of the signal light detected by the color of the signal light, a position of the image in the image, And a traffic light state classifying unit for classifying the presence or state of the traffic light based on the boundary region of the traffic lights.

Description

[0001] APPARATUS AND METHOD FOR RECOGNITION SIGNAL LIGHT USING MULTIPLE EXPOSURE IMAGE OF CAMERA [0002]

The present invention relates to an apparatus and method for automatically recognizing a traffic light using multiple exposure images continuously shot with different exposure times of a camera.

Generally, the recognition of the traffic lights using camera images is performed through two processes of detection and classification.

The detection process recognizes the colors of the traffic lights such as red, yellow, and green using the color information. At this time, in order to recognize a color, a color range to be detected is set in a specific color space, and it is determined whether or not the color information of the input image exists in the previously set color range. Therefore, different color ranges must be set to detect the red, yellow, and green of the traffic light, respectively. However, since the camera image is expressed in the RGB color space, the RGB color space exhibits nonlinear changes in various lighting conditions, so it is not easy to detect the color.

To compensate for these drawbacks, the HSV color space or the Normalized RGB color space has been applied recently according to a number of studies. In particular, the HSV color space independently represents hue, saturation, and brightness (brightness or value). In this method, since all the color information is determined by the hue, the color range for detecting a specific color can be defined in the hue. In addition, since the illumination change is represented by the value, the nonlinearity of the RGB color space can be compensated. For example, in order to detect red, yellow, and green of the traffic lights, each color range is set in the HSV color space and the input image is discriminated.

However, the above-described detection process using the HSV color space also has two problems.

First, the traffic light color saturates in the image. For normal cameras, use the auto exposure function to maintain the image balance. The auto exposure function adjusts the exposure time of the camera by calculating the average brightness of the image to prevent the image from becoming too dark or bright. The auto exposure function keeps the exposure time longer in low light conditions. As the exposure time becomes longer, the traffic light color becomes saturated in the image. For example, in FIG. 1, red light appears to be yellow due to color saturation phenomenon. To solve this problem, there is an additional problem that if the color range for detecting red is widened, it is impossible to distinguish between red and yellow.

Second, there is a possibility that the detection error will increase when using the general exposure image of the camera. That is, when the vehicle is traveling in the city center, there are many similar colors other than the traffic lights in the general exposure image, which causes the false detection. Such an erroneous detection directly affects the signal light recognition, which leads to a deterioration in recognition performance as a whole.

On the other hand, in the classification process, color information obtained in the detection process is used to classify the presence or the state of a traffic light. Specific classification methods include using geometric information of traffic lights, using machine learning such as Adaboost, and using map information.

However, the method using existing geometric information is a method of comparing the black color region of the traffic light with the detected color locus, and there is a problem that the black color region must be detected definitely. In addition, the machine learning method using Adaboost uses the Haar-like feature, but it has a disadvantage that it is difficult to find a representative feature because it has a simple structure in the case of a traffic light. In addition, there is an additional burden of generating a map besides image recognition in the method of utilizing the map information.

Related prior arts include Korean Patent Laid-Open Publication No. 10-2009-0055848 (name of the invention: apparatus and method for recognizing a traffic light, publication date: June 03, 2009).

SUMMARY OF THE INVENTION An object of the present invention is to provide a method and apparatus for detecting a color of a traffic light installed at an intersection or crosswalk by capturing a multiple exposure image from a camera and comparing the multiple exposure images to classify the presence or the status of the traffic light based on the acquired color information, And an apparatus and method for automatically recognizing a traffic light using a multiple exposure image of a camera which can accurately recognize the traffic light.

The above-mentioned object is achieved by an image pickup apparatus comprising: a photographing unit photographing a front of a vehicle; A multiple exposure image capturing unit for adjusting exposure time of the photographing unit and capturing a multiple exposure image continuously photographed through the photographing unit according to an exposure time; A signal light color detector for detecting a color of a traffic light by analyzing a color distribution of a multiple exposure image acquired through the multiple exposure image capture unit; And a traffic light state classifying unit for setting a boundary region of a traffic light based on the color, size, and position of the traffic light detected through the traffic light color detecting unit and distinguishing the presence or state of the traffic light based on the boundary region of the traffic light The present invention relates to an automatic recognition device for a traffic light using a multiple exposure image of a camera.

Wherein the multiple exposure image capturing unit sets an exposure time of the photographing unit to a normal exposure mode and a low exposure mode that is darker than the general exposure mode; When the signal lamp recognition operation is started, the photographing section is instructed to continuously take photographs in accordance with the low-exposure mode and the general exposure mode set through the exposure time control section, and the low-exposure image and the general exposure image successively photographed according to the mode are captured And an image capturing control unit.

The exposure time controller may measure the intensity of the normal exposure image so as to balance the image in the normal exposure mode and adjust the exposure time according to the intensity of the light.

The traffic light color detecting unit may detect the color of the traffic light based on the color distribution of the low-exposure image among the multiple exposure images.

The traffic light state classifying unit may set a boundary region of the traffic light based on the color, size, and position of the traffic light detected by the traffic light color detecting unit in the general exposure image among the multiple exposure images.

The traffic light state classifying unit may classify the presence or the state of a traffic light by applying a classification algorithm based on a histogram of an Oriented Gradient (HOG) to the boundary region of the traffic lights.

The traffic light status classifier may compare the status of the traffic lights classified on the basis of the classification algorithm with the data stored in the database to recognize the traffic lights.

The object of the present invention is achieved by a method for automatically recognizing a traffic light, comprising the steps of: photographing a front part of a vehicle according to an exposure time of a predetermined camera when an operation start for recognizing a traffic light is inputted; Capturing a multiple exposure image captured continuously by the photographing unit in a multiple exposure image capturing unit of the automatic traffic light recognition apparatus; Detecting a color of a traffic light by analyzing a color distribution of a low-exposure image among the multiple exposure images in a traffic light color detection unit of the automatic traffic light recognition apparatus; A step of setting a boundary region of a traffic light based on the color, size, and position of the traffic light detected through the traffic light color detection unit with respect to a general exposure image among the multiple exposure images in the traffic light state classification unit of the automatic traffic light recognition apparatus ; And distinguishing the presence or the state of the traffic light based on the boundary region of the traffic lights in the traffic light state classification unit of the automatic traffic light recognition apparatus.

According to an embodiment of the present invention, it is possible to solve the color saturation phenomenon that has occurred in the signal light recognition method using the existing camera and accurately recognize the color of the traffic light, thereby preventing the unrecognition of the traffic light.

According to an embodiment of the present invention, since a low-exposure image among multiple exposure images captures only a portion of a traffic light in which brightness is strong, it is possible to reduce an erroneous detection rate due to a similar color and to prevent an accident caused by a misunderstanding of a traffic light.

Also, according to the embodiment of the present invention, it is possible to support safe driving of the driver by applying to an intelligent vehicle or a driver assistance system.

FIG. 1 is a view showing a color saturation phenomenon caused by a conventional automatic signal light recognition apparatus.
FIG. 2 is a block diagram for explaining a signal light automatic recognition apparatus according to an embodiment of the present invention.
FIG. 3 is a block diagram showing a detailed configuration of the multiple-exposure image capturing unit shown in FIG.
4 is a view for explaining exposure time and capture timing for implementing a multiple exposure image according to an embodiment of the present invention.
FIG. 5 is a block diagram illustrating an exposure time control operation for implementing a multiple exposure image according to an embodiment of the present invention. Referring to FIG.
FIG. 6 is a diagram illustrating a multiple exposure image and a color distribution captured through an automatic traffic light recognition apparatus according to an embodiment of the present invention.
7 is a view for explaining a method of setting a boundary region of a traffic light in a multiple exposure image captured through an automatic traffic light recognition apparatus according to an embodiment of the present invention.
8 is a diagram showing an example of distinguishing the state of the traffic lights in the traffic light condition classifying section shown in Fig.
9 is a flowchart illustrating a method of recognizing a traffic light according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and / or features of the present invention, and how to accomplish them, will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but is capable of many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram illustrating an apparatus for automatically recognizing a traffic light according to an exemplary embodiment of the present invention. Referring to FIG.

The automatic signal light recognition apparatus 100 according to an exemplary embodiment of the present invention includes a photographing unit 110, a multiple exposure image capturing unit 120, a traffic light recognizing unit 150, and a database 160. The signal lamp recognition unit 150 may include a signal lamp color detector 130 and a signal lamp classifier 140 as shown in the figure.

In this embodiment, the photographing section 110 photographs the front of the vehicle. The photographing unit 110 may be installed in a vehicle and may be photographed in a state of being carried by a user in the vehicle.

Also, at least one photographing unit 110 may be used, and a camera having a high frame rate may be used. A high frame rate means that the rate at which the camera shoots successive images is high.

In addition, the photographing unit 110 can be installed in the vehicle so as to be less influenced by outside weather when the photographing unit 110 is installed in the vehicle, and can also be installed outside the vehicle.

The multiple exposure image capturing unit 120 adjusts the exposure time of the photographing unit 110 and captures the multiple exposure images continuously photographed according to the preset exposure time from the photographing unit 110. [ At this time, the multiple exposure images are images having different exposure times, and it is assumed that they are photographed at the same time although there is a minute time difference due to continuous shooting (assuming time consistency). For example, in the case of two images, one can be captured in Normal Exposure mode and the other in Capture mode in Low Exposure mode.

As shown in FIG. 3, the multi-exposure image capturing unit 120 includes an image capturing control unit 122, an exposure time control unit 124, and the like.

The exposure time control unit 124 sets the exposure time of the photographing unit 110 to a normal exposure mode and a low exposure mode that is darker than the normal exposure mode. Accordingly, when the operation for recognizing the traffic light is started, the exposure time control unit 124 sets the exposure time of the photographing unit 110 to the predetermined normal exposure mode and the low exposure mode, and controls the photographing.

As shown in FIG. 4, the low exposure mode has a short exposure time (Exposure Time). This is done by using an active light source in which the traffic lights themselves emit light. Therefore, even if the exposure time is fixed short, it is possible to capture the image of the traffic light color because it emits light to the traffic light itself. The image captured in the low exposure mode (hereinafter referred to as a low-exposure image) is characterized in that the overall brightness of the image is darker than the image captured in the general exposure mode (hereinafter referred to as a general exposure image).

The normal exposure mode is set to the exposure value for normal shooting. For example, when the exposure value can be set in the (+) direction or (-) direction based on the average value of 0, the normal exposure mode may be set to the average value of 0, and the low exposure mode may be set to the range of the negative direction.

Since the normal exposure mode may be difficult to capture a clear image due to the external environment such as the weather or the night, the exposure time can be adjusted so as to balance the image. The adjustment is performed by the exposure time control unit 124.

That is, as shown in FIG. 5, the exposure time control unit 124 measures the intensity of a general exposure image photographed through a camera so as to balance the image in the normal exposure mode, You can adjust the exposure time. If the light intensity of a general exposure image photographed through a camera is higher than the average, the exposure time may be adjusted so as to lower the light intensity so that the normal exposure image can be photographed.

3, the image capturing control unit 122 of the multiple exposure image capturing unit 120 may be configured to transmit the image capturing control information to the image capturing unit 110 through the exposure time control unit 124, Shoot continuously according to the exposure mode and normal exposure mode. Then, the low-exposure image and the general exposure image successively photographed from the photographing unit 110 are captured. Accordingly, the two captured images through the multiple exposure image capturing unit 120 are represented as shown in FIG. 6 (a) is a general exposure image and (b) is a low-exposure image.

Referring to FIG. 6 (b), it can be seen that there is no color saturation phenomenon which has been pointed out as a problem in the low-exposure image, and only the signal light which is an active light source appears clearly. In addition, in the case of a low-exposure image, only a part having a strong light intensity such as an active light source is captured, thereby preventing a false detection rate of a similar color.

2, the traffic light recognizing unit 150 of the automatic traffic light recognizing apparatus 100 will be described.

The signal color detection unit 130 of the signal light recognition unit 150 analyzes the color distribution of the low-exposure image captured through the multiple-exposure image capture unit 120 to detect the color of the signal light.

For example, FIG. 6 shows an example of analyzing the color distribution of an image. The hue range of the general exposure image (a) is widely distributed, whereas the low exposure image (b) is very narrowly distributed compared with the color distribution of the general exposure image. This is effective for detecting one color such as red, yellow, and green.

7, the traffic light state classifying unit 140 sets a boundary box (P1, P2 in FIG. 7) of a traffic light in a general exposure image captured through the multiple exposure image capturing unit 120 And a support vector machine (SVM) based on a histogram of an Oriented Gradient (HOG) is applied to the boundary regions (P1 and P2 in FIG. 7) of the set traffic lights to discriminate the presence or the status of the traffic lights . That is, whether or not the image existing in the boundary region (P1 and P2 in FIG. 7) of the signal lamp is correct, what kind of signal is present, whether the current state of the signal or the like is a left turn, .

At this time, the border area setting of the traffic lights can be performed based on the color, size, location in the image, etc. of the traffic light detected through the traffic light color detecting unit 130. [

The gradient histogram (HOG) is a feature descriptor representing a feature of an image by expressing a brightness change in an image as a histogram. In the case of a traffic light, it is easy to extract the feature using the gradient histogram (HOG) because the black area of the traffic light has a clear brightness change with the surrounding background and the position of the sphere in the traffic light changes according to the state of the traffic light.

The database 160 stores information on classification algorithms based on the gradient histogram (HOG), types of traffic lights (2, 3, 4, and so on), status information, and the like. Accordingly, the data stored in the database 160 can be provided as reference data when the traffic light state classifier 140 identifies the state of the traffic lights.

FIG. 8 shows an example of the result of classification algorithm analysis according to six states of a three-way / four-way traffic light.

In SVM (Support Vector Machine), since classifier is created through machine learning, HOG Descriptor is extracted by using multiple sample images representing states of each traffic light, and then used in classifier learning. The generated classifier can classify the image of the boundary region input in real time into six states.

9 is a flowchart illustrating a method of recognizing a traffic light according to an embodiment of the present invention.

Referring to each step of the flowchart in conjunction with the components of FIG. 2, in the first step S1, the automatic signal recognizing apparatus 100 confirms whether an operation for recognizing a traffic light is requested through the input unit.

After the confirmation, if the operation for recognizing the traffic lights is not requested in the step S2, another operation is performed.

If it is determined in step S3 that an operation for recognizing a traffic light is requested, the photographing unit 110 of the automatic traffic light recognition apparatus 100 operates the camera and performs continuous photographing according to a predetermined exposure time.

Thereafter, in step S4, the multiple exposure image capturing unit 120 of the automatic signal recognition device 100 captures a multiple exposure image from an image continuously captured through the photographing unit 110. [ Here, multiple exposure images have different exposure times and are considered to be photographed at the same point in time.

The step (S4) may specifically include the following steps.

In the multi-exposure image capturing unit 120 of the automatic traffic light recognition apparatus 100, the exposure time of the camera is set to a normal exposure mode, a dark exposure mode that is darker than the general exposure mode, Continuous shooting is instructed according to low exposure mode and normal exposure mode. Then, the multiple exposure image capturing unit 120 of the automatic traffic light recognition apparatus 100 captures the low-exposure image and the general exposure image captured by the photographing unit 110, respectively.

Next, in steps S5 and S6, the multiple exposure image capturing unit 120 of the automatic signal recognition device 100 may adjust the exposure time of the camera when retrying the capture of the multiple exposure image. In particular, in the normal exposure mode, the exposure time can be adjusted according to the measured light intensity by measuring the intensity of a general exposure image photographed through a camera so as to balance the image.

Thereafter, in step S7, the traffic light color detector 130 of the automatic traffic light recognition apparatus 100 analyzes the color distribution of the low-exposure image among the multiple exposure images to detect the color of the traffic light.

In step S8, the traffic light state classifying unit 140 of the automatic traffic light recognition apparatus 100 sets the boundary region of the traffic lights in the general exposure image among the multiple exposure images and sets the boundary region of the traffic lights set in the gradient histogram Based classification algorithm to classify the presence or status of traffic lights.

The boundary region of the traffic lights can be set based on the color, size, and position in the image of the traffic light obtained from the low-exposure image. It is necessary to apply classification algorithm based on the boundary area of the traffic lights because it is difficult to judge whether or not the state of an accurate traffic light is obtained only with the color information of the traffic light.

Finally, in step S9, the automatic traffic light recognition apparatus 100 compares the color and state information of the detected traffic lights with the data stored in the database 160 to recognize the traffic lights.

Thus, one embodiment of the present invention proposes two new methods for signal light recognition. The first proposal is an image capturing method using multiple exposures, and the second proposal is a signal light state classification method using a support vector machine (SVM) based on a histogram of oriented gradients (HOG).

In other words, the signal light color was detected in the low exposure image captured through the multiple exposure method, and the recognition rate of the traffic light was improved by classifying the state of the traffic light as the general exposure image. In addition, by applying the gradient histogram (HOG) based SVM classifier to the traffic light status classification process, it provides better performance than the existing classification method. Therefore, according to the embodiment of the present invention, a traffic light installed at an intersection or a crosswalk can be accurately recognized by a camera, so that an accident caused by a traffic light unrecognized or a false recognition can be prevented.

Embodiments of the present invention include a computer readable medium having program instructions for performing various computer implemented operations. The computer-readable medium may include program instructions, local data files, local data structures, etc., alone or in combination. The media may be those specially designed and constructed for the present invention or may be those known to those skilled in the computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and ROMs, And hardware devices specifically configured to store and execute the same program instructions. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Modification is possible. Accordingly, the spirit of the present invention should be understood only in accordance with the following claims, and all equivalents or equivalent variations thereof are included in the scope of the present invention.

100: Automatic recognition of traffic lights
110:
120: multiple exposure image capturing unit
130: traffic light color detection unit
140: Traffic light state classification section
150: signal lamp recognition unit
160: Database
122: image capture control unit
124: Exposure time control section

Claims (8)

A photographing unit photographing the front of the vehicle;
A multiple exposure image capturing unit for adjusting exposure time of the photographing unit and capturing a multiple exposure image continuously photographed through the photographing unit according to an exposure time;
A signal light color detector for detecting a color of a traffic light by analyzing a color distribution of a multiple exposure image acquired through the multiple exposure image capture unit; And
And a traffic light state classifying unit for setting a boundary region of the traffic light based on the color, size, and position of the traffic light detected by the traffic light color detecting unit, and discriminating the presence or the state of the traffic light based on the boundary region of the traffic light ,
Wherein the multiple exposure image capture unit comprises:
An exposure time controller for setting the exposure time of the photographing unit to a normal exposure mode and a low exposure mode which is darker than the general exposure mode; And
When the signal lamp recognition operation is started, the photographing unit sequentially instructs the photographing unit to alternately take a low-exposure mode and a general exposure mode set through the exposure time control unit, and sequentially displays a low- And an image capturing control unit for capturing images, respectively,
The traffic light color detecting unit detects the color of the traffic light based on the color distribution of the image with respect to the low-exposure image among the multiple exposure images,
The traffic light condition classifier sets a boundary region of the traffic light based on a general exposure image among the multiple exposure images, and based on the color of the traffic light detected from the low-exposure image and the boundary region between the traffic light set from the general exposure image, The presence or absence of a traffic light is distinguished by applying a classification algorithm based on a histogram of an Oriented Gradient (HOG) using a change in brightness in an image with respect to a boundary region of the traffic light,
Wherein the classification algorithm extracts a gradient histogram feature (HOG Descriptor) using a plurality of sample images representative of the states of the traffic lights, and then uses the same to classifier learning.
delete The method according to claim 1,
The exposure time control unit
Wherein the light intensity of the general exposure image is measured to adjust the balance of the image in the normal exposure mode and the exposure time is adjusted according to the light intensity.
delete delete delete The method according to claim 1,
Further comprising a database storing information on the type of the signal lamp including the second, third, and fourth spheres and information on a state of a signal indicating whether the signal is a left turn, a right or a straight signal,
Wherein the traffic light condition classifier compares the status of the traffic lights classified on the basis of the classification algorithm with the data stored in the database to recognize the traffic lights.
Photographing the front of the vehicle in accordance with the exposure time of the predetermined camera when an operation start for recognizing the traffic light is inputted in the photographing part of the automatic traffic light recognition device;
Capturing a multiple exposure image captured continuously by the photographing unit in a multiple exposure image capturing unit of the automatic traffic light recognition apparatus;
Detecting a color of a traffic light by analyzing a color distribution of a low-exposure image among the multiple exposure images in a traffic light color detection unit of the automatic traffic light recognition apparatus;
A step of setting a boundary region of a traffic light based on the color, size, and position of the traffic light detected through the traffic light color detection unit with respect to a general exposure image among the multiple exposure images in the traffic light state classification unit of the automatic traffic light recognition apparatus ; And
And a step of distinguishing whether or not a traffic light is present based on a boundary region of the traffic light in a traffic light state classification unit of the automatic traffic light recognition apparatus,
In the capturing of the multiple exposure image,
Wherein the multiple exposure image capturing unit sets the exposure time of the photographing unit to a low exposure mode that is darker than the general exposure mode and the general exposure mode, and alternately sets the low-exposure mode and the general exposure mode, And captures the low-exposure image and the general exposure image successively photographed according to the mode, respectively,
In the step of detecting the color of the traffic light, the traffic light color detecting unit detects the color of the traffic light on the basis of the color distribution of the image with respect to the low-exposure image among the multiple exposure images,
In the step of distinguishing the state of the traffic lights, the traffic light state classifying unit sets the boundary region of the traffic lights based on the general exposure image among the multiple exposure images, and sets the boundary between the color of the traffic lights detected from the low- The present invention relates to a method of classifying the state of a traffic light based on a boundary region of a traffic light and applying a gradation histogram based on a histogram of an Oriented Gradient (HOG) However,
Wherein the classification algorithm extracts a gradient histogram feature (HOG Descriptor) using a plurality of sample images representative of the states of the signal lights, and then uses the extracted histogram features for classifier learning.

Images