KR20200101509A - Method and apparatus for detecting traffic light - Google Patents

Abstract

Disclosed is a traffic light detecting method. According to one embodiment of the present invention, the traffic light detecting method comprises: an image conversion step in which a first learning model converts an input image photographed of a traffic light and a surrounding environment thereof into a low-channel converted image composed of at least a first signal color, a second signal color, and a background color; a candidate region selection step of selecting at least one traffic light candidate region based on a plurality of first color groups having the first signal color and a plurality of second color groups having the second signal color in the low-channel converted image by a second learning model; and determining, by a third learning model, a state of the traffic light based on partial images of the input image corresponding to the traffic light candidate region. According to another embodiment of the present invention, provided is a traffic light detection device which performs traffic light detection by the traffic light detection method.

Description

Method and apparatus for detecting traffic light TECHNICAL FIELD

The present disclosure relates to a traffic light detection method and apparatus.

The content described in this section merely provides background information on the present disclosure and does not constitute prior art.

Deep learning is a kind of machine learning, and refers to using multiple nonlinear processing layers by learning representative features that are useful directly from data. Meanwhile, the necessity of developing technology to recognize and analyze objects based on image data collected from smartphones, CCTV, black boxes, satellite images, etc. using machine learning or deep learning, is being emphasized. .

A technology for real-time recognition of surrounding road conditions is being developed for autonomous vehicle driving. In particular, this technology detects in real time events that occur when a vehicle is driving on the road, for example, the driving state of adjacent vehicles, whether an accident has occurred, and the road condition, and adjusts the driving state of the vehicle in response to the detected event. It is necessary to inform the driver.

However, the road conditions around the vehicle not only continuously change according to the location of the road and the driving time, but also the data on the external environment captured by the camera are also weather, the direction of the sun, and the driving time (day or night). Since it is dependent, it is not easy to reliably detect the surrounding situation of the vehicle, particularly the location or state of traffic lights, taking into account all of these dynamic variables.

Accordingly, attempts are being made to apply the aforementioned deep learning to the recognition of the surrounding situation of the vehicle, in particular, detection of traffic lights.

However, for the detection of traffic lights and their follow-up measures, the detection of traffic lights must be reliably carried out even at a considerable distance, for example 50 to 100 meters from the traffic light. Since the detection of a traffic light is performed through external environment data recognized through a camera, that is, image data, a high-resolution video image is required to identify a traffic light from a considerable distance. When the resolution of the video image used in the process of identifying a traffic light increases, the amount of computation for detecting a traffic light increases, which may reduce the real-time performance of the traffic light recognition. In addition, this causes difficulty in increasing the processing speed or cost of the system. .

In addition, the area or number of pixels occupied by an image corresponding to a traffic light in the entire surrounding image having a high resolution is relatively small. This can be interpreted as having little effect on the characteristics of the entire image, especially the change in the color of the traffic light. That is, it is a difficult problem to learn a reliable and reliable traffic light detection model that is sensitive to changes in the location and state of the traffic light.

Accordingly, in recognizing the location and state of traffic lights on the road during vehicle driving, the present invention improves applicability to an embedded system by optimizing the traffic light recognition model, and minimizes the amount of computation to secure real-time when applying a real vehicle, and The main purpose is to provide the device.

In addition, an object of the present invention is to provide a traffic light detection method and apparatus capable of increasing the recognition accuracy of a traffic light through supervised learning.

Objects of the present invention are not limited to the problems mentioned above, and other objects that are not mentioned will be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention, an image conversion step of converting, by the first learning model, an input image photographing a traffic light and its surrounding environment into a low-channel converted image composed of at least a first signal color, a second signal color, and a background color, A candidate region in which the second learning model selects one or more traffic light candidate regions based on a plurality of first color groups having the first signal color and a plurality of second color groups having the second signal color in the low-channel converted image It provides a traffic light detection method including a selection step, and a traffic light state determination step of determining a state of the traffic light based on partial images of the input image corresponding to the traffic light candidate region by a third learning model.

According to another embodiment of the present invention, there is provided a traffic light detection apparatus for detecting a traffic light by the above traffic light detection method.

1 is a flow chart showing each process performed in a traffic light detection method and apparatus according to an embodiment of the present invention.
2 is a conceptual diagram illustrating each process performed in a method and apparatus for detecting a traffic light according to an embodiment of the present invention.
3 is a flow chart showing a detailed process of an image conversion step according to an embodiment of the present invention.
4 shows an exemplary input image according to an embodiment of the present invention.
5 shows an exemplary low-channel converted image according to an embodiment of the present invention.
6 is a flow chart showing a learning process of a first learning model in an embodiment of the present invention.
7 is a flow chart showing a detailed process of the step of selecting a candidate region according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating color group correspondence regions corresponding to a plurality of color groups on an input image according to an embodiment of the present invention.
9 is a diagram illustrating a state in which a traffic light candidate area is displayed on an input image according to an embodiment of the present invention.
10 is a flow chart showing a learning process of a second learning model in an embodiment of the present invention.
11 is a flow chart showing a detailed process of the step of determining a state of a traffic light according to an embodiment of the present invention.
12 illustrates a state in which a traffic light state of a traffic light candidate region is detected in a traffic light state determination step according to an embodiment of the present invention.
13 is a flow chart illustrating a learning process of a third learning model in an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding reference numerals to constituent elements in each drawing, it should be noted that the same constituent elements are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

In describing the components of the embodiment according to the present invention, reference numerals such as first, second, i), ii), a), b) may be used. These codes are only for distinguishing the constituent elements from other constituent elements, and the nature, order, or order of the constituent elements are not limited by the symbols. In the specification, when a part'includes' or'includes' a certain element, it means that other elements may be further included rather than excluding other elements unless explicitly stated to the contrary. .

1 is a flowchart showing each process performed by a traffic light detection method and apparatus according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating each process performed by a traffic light detection method and apparatus according to an embodiment of the present invention. It is a conceptual diagram.

Hereinafter, each process performed in a traffic light detection method and apparatus according to an embodiment of the present invention will be schematically described with reference to FIGS. 1 and 2.

Referring to FIG. 1, a method of detecting a traffic light according to an embodiment of the present invention includes an image conversion step (S1), a candidate region selection step (S2), and a traffic light state determination step (S3).

In the image conversion step S1, the input image 11 is converted into a low-channel converted image 12 having a plurality of channels.

In this case, the input image 11 refers to an external image of the vehicle, in particular, an image photographed and acquired by a camera installed to photograph the front of the vehicle. Here, the size, installation location, and type of the camera are not limited, and for example, it may be an image obtained from a black box camera while driving a vehicle.

For example, photographing for obtaining the above input image 11 may be performed in real time while the vehicle is driving.

On the other hand, for example, in a general photographed image, for example, an image having Full HD quality, when a traffic light color sphere, which is a signal lighting bulb of a traffic light, occupies approximately 10 pixels or more, it is possible to perform effective recognition of the traffic light color sphere.

At this time, it was confirmed that the distance between the traffic light and the vehicle was about 103 meters. Therefore, for example, from the moment the distance between the traffic light and the vehicle is within about 103 meters, it may be possible to recognize a traffic light, specifically a color sphere of a traffic light.

However, this is only an example, and the size of the distance at which the vehicle can recognize a traffic light may vary according to an increase or decrease in image quality of a captured image.

Meanwhile, the input image 11 includes not only information on traffic lights, but also information on other vehicles on the road and surrounding environments. Therefore, it is important to discriminate only information about pure traffic lights (to be precise, color spheres of traffic lights), excluding information about other vehicles on the road and surrounding environment on the input image 11.

In order to distinguish such information, unlike an embodiment of the present invention, when the color of each element constituting the input image 11 is close to the color of the traffic light by more than a specific threshold, without using a learning model, You can think of a way to judge it as consistent with.

However, when the traffic light colors are classified in this way, the detection performance may be significantly deteriorated because it is sensitive to various environmental changes (illumination, driving environment). On the other hand, methods based on artificial intelligence learning models receive the entire image as an input and have to extract various features through the base network, so the number of model parameters on the base network increases considerably. In this case, approximately 7 million or more of parameters are required, and accordingly, the amount of computation that must be performed to recognize the state of a traffic light increases.

This increases the complexity of the network for calculation, and also increases the calculation time for recognizing the state of a traffic light. This is not appropriate to determine the state of traffic lights in a short time in real time while driving on the road.

Accordingly, the present invention proposes a method of reducing parameters for calculation by using a supervised learning method instead of using a method of classifying the color on the input image 11 based on a specific threshold in recognizing a traffic light.

That is, according to an embodiment of the present invention, in the image conversion step S1, the input image 11 is inputted to the first learning model 10 configured with a neural network and converted into a low-channel converted image 12.

Here, the first learning model 10 is composed of a neural network trained in a deep learning method as described in detail later.

In addition, the input image 11 is converted into a low-channel transformed image 12 using the first learning model 10, which has already completed learning through a plurality of training data, which reduces the number of traffic light candidates. In addition, since the learning model composed of fewer parameters is used, the amount of computation can be significantly reduced.

According to an embodiment of the present invention, by using such a method, parameters for calculation in the image conversion step (S1) may be exemplarily limited to less than one hundred.

On the other hand, the low-channel converted image 12 refers to an image converted so that the received input image 11 is configured with only a limited channel by passing through the image conversion step S1. Specifically, the low-channel converted image 12 is an image composed of pixels having a limited type of color.

This is due to the fact that the color of traffic lights is very limited, such as usually red or green. That is, since the signal color of the traffic light is very limited in its type, in the image conversion step S1, all information on the color that does not correspond to the signal color of the traffic light is removed so that unnecessary calculations are not performed in a later step.

Meanwhile, in this case, the plurality of channels constituting the low-channel converted image 12 may be channels respectively corresponding to each signal color of a traffic light, as will be described in more detail later.

The plurality of channels constituting the low-channel converted image 12 are illustratively a first channel 13 corresponding to a first signal color of a traffic light, a second channel 14 corresponding to a second signal color of a traffic light, and others. It may be composed of a third channel 15 corresponding to the color.

Here, as an example, the first signal color of the traffic light may be designated as red, and the second signal color may be designated as green, and colors other than red and green may be designated as other colors.

In the candidate region selection step (S2), a plurality of color groups having a color corresponding to a signal color of a traffic light are detected on the low-channel converted image 12, and a traffic light candidate region 17 (see FIG. 9) based on the plurality of color groups. Screen.

Meanwhile, the color group here refers to an aggregate in which a plurality of pixels having similar colors are adjacent and clustered. For example, an area illuminated in red on a traffic light on the input image 11 may be expressed as a color group formed of red pixels arranged adjacent to each other in a circular shape as a whole on the low-channel converted image 12.

In this case, the traffic light candidate region 17 may be a term that refers to a region on the input image 11 corresponding to a color group whose probability of corresponding to an actual traffic light is greater than or equal to a certain probability among the plurality of color groups.

In the candidate region selection step (S2), the traffic light candidate region 17 is selected using the second learning model 20 as described in detail later. In this case, like the first learning model 10, the second learning model 20 may be a learning model composed of a neural network that has been previously learned by deep learning.

In the traffic light state determination step (S3), based on the traffic light candidate region 17 selected in the candidate region selection step (S2), the state of the traffic light corresponding to the corresponding region is determined. At this time, the determined state of the traffic light may include a location of the traffic light and a color or direction indication of the traffic light.

Specifically, in the traffic light state determination step (S3), the traffic light state result data 18, which is analysis data on the state of the traffic light, may be output. At this time, the traffic light status result data (18) is the detection value (o) for the existence of the actual traffic light, the detection value (s ₁ , s ₂ , s ₃ , s ₄ ) for the status of the traffic light, and the location and size of the traffic light. It may include detection values (x _c , y _c , w, h).

Here, the detected value for the traffic light status may be a value related to the color or shape of the signal lit on the traffic light, and the detected value for the position and size of the traffic light is a coordinate value for the position of the traffic light (x _c , y _c ) or a traffic light candidate. It may be the horizontal and vertical lengths (w, h) of the region.

In the traffic light state determination step (S3), the state of the traffic light is determined using the third learning model 30, as will be described in detail later. In this case, the third learning model 30 may be a learning model composed of a neural network previously learned by deep learning, like the first learning model 10 and the second learning model 20.

3 is a flow chart showing a detailed process of an image conversion step according to an embodiment of the present invention.

In addition, FIG. 4 illustrates an exemplary input image according to an embodiment of the present invention, and FIG. 5 illustrates an exemplary low-channel converted image according to an exemplary embodiment of the present invention.

Hereinafter, a detailed process of the image conversion step S1 in an embodiment of the present invention will be described with reference to FIGS. 3 to 5.

First, an input image 11 photographing a traffic light and a surrounding environment when driving a vehicle, exemplarily illustrated in FIG. 4, is obtained. For example, photographing for obtaining the above input image 11 may be performed in real time while the vehicle is driving.

After acquiring the input image 11, the input image 11 is input to the first learning model 10. At this time, the first learning model 10 is a learning model composed of a Convolutional Neural Network (CNN), and is a model that has been previously trained to convert the input image 11 into a low-channel transformed image 12 when driving a vehicle. At this time, the CNN constituting the first learning model 10 will be referred to as a first CNN.

The input image 11 input to the first learning model 10 is converted into a low-channel converted image 12 composed of a plurality of channels. At this time, the low-channel converted image 12 is a low-channel converted image 12 having a limited number of channels. Also, the above channels are composed of channels corresponding to a plurality of signal colors of a traffic light.

For example, since the most common traffic lights are configured to have a red or green signal, the low-channel converted image 12 includes a first channel 13 corresponding to a red image element and a second channel 14 corresponding to a green image element. And, a third channel 15 corresponding to an image element with an index background color other than red and green.

In addition, as an example, if the first signal color of the traffic light is red and the second signal color is green, the first signal color on the low-channel converted image 12 shows the first color group and the second signal color clustering in a certain area. A second color group forming a cluster in a certain area may be formed.

Meanwhile, the low-channel converted image 12 may be a 4-channel image instead of the above three-channel image.

In this case, each channel constituting the low-channel converted image 12 may further include a channel corresponding to a yellow image element as well as a channel corresponding to a red, green, and background color image element. This is a configuration reflecting the fact that traffic lights on real roads have not only red and green signals, but also yellow signals.

As described above, the process of converting the input image 11 into the low-channel transformed image 12 through the first learning model 10 composed of the first supervised learning CNN is very small, so the state of the traffic light is detected. You can perform very fast calculations in the process.

6 is a flow chart showing a learning process of a first learning model in an embodiment of the present invention. Hereinafter, a learning process of the first learning model 10 in an embodiment of the present invention will be described with reference to FIG. 6.

The first learning model 10 is composed of a first CNN, and may be learned by a supervised learning method.

First, secure multiple images. In this case, the plurality of images may be a plurality of images, such as the input image 11 according to an embodiment of the present invention, which captures a traffic light on a road existing in several places and its surrounding environment.

In addition, data indicating information on the positions of traffic lights on a plurality of secured images is accumulated.

At this time, the information on the traffic light is information on the location of the traffic light directly displayed by a person with reference to the plurality of images above.

Meanwhile, a set of data displaying traffic light information on a plurality of images thus secured will be collectively referred to as first training data.

In addition, by securing the low-channel converted image 12, a plurality of signal color regions and background color regions expressed on the low-channel converted image 12 are secured. In this case, the low-channel converted image 12 may be, for example, the low-channel converted image 12 obtained by performing the image conversion step S1 in the embodiment of the present invention.

In addition, the first learning model 10 is trained based on the plurality of signal color regions, background color regions and first training data expressed on the secured low-channel converted image 12. That is, by comparing the low-channel transformed image 12 generated by the first learning model 10 and the first training data corresponding to a kind of correct answer, it is determined whether or not a traffic light exists on the low-channel transformed image 12. The first learning model 10 is learned by evaluating based on one training data.

In this case, error optimization may be performed in the learning process of the first learning model 10. This is to ensure that the area corresponding to the position of the traffic light is reflected faithfully and without errors on the low-channel converted image 12 converted in the image conversion step (S1).

For example, when the low-channel converted image 12 output in the learning process of the first learning model 10 is compared with the first training data, the low-channel converted image 12 does not correspond to the first training data. Error may exist.

At this time, the possibility that the first learning model 10 will omit the area corresponding to the traffic light area by performing error optimization in which weight is applied only to the error value related to the area corresponding to the traffic light area among the low-channel transformed images 12. Can lower.

Specifically, a weight assigned only to an error value related to an area corresponding to the traffic light area may be set to be 1000 times or more. This is because the proportion of the area corresponding to the traffic light area in the entire image is very small. Therefore, by assigning a weight of 1000 times or more as described above, learning of the first learning model 10 can be effectively performed.

On the other hand, at this time, learning may be conducted to increase the recognition rate of a color sphere of a traffic light through a loss function of an appropriate form.

7 is a flow chart showing a specific process of the candidate region selection step (S2) according to an embodiment of the present invention.

In addition, FIG. 8 shows a color group-corresponding area corresponding to a plurality of color groups on an input image in an embodiment of the present invention, and FIG. 9 shows a traffic light candidate area on an input image in an embodiment of the present invention. It shows the state.

Here, the color group correspondence region 16 refers to an area on the input image 11 corresponding to the color group formed on the low-channel converted image 12, and the traffic light candidate region 17 is an input among a plurality of color group correspondence regions 16. It refers to an area determined to correspond to the location of a traffic light on the image 11.

On the other hand, in FIG. 8, for convenience, only a part of the color response region 16 is indicated with reference numerals, but it is noted that all of the separately marked regions on the input image 11 correspond to the color group correspondence region 16.

Hereinafter, a process of selecting a candidate region (S2) in an embodiment of the present invention will be described in detail with reference to FIGS. 7 to 9.

First, information on the low-channel converted image 12 converted in the image conversion step S1 is input into the second learning model 20.

In this case, exemplary information input to the second learning model 20 is geometric information, such as positions and sizes of a plurality of color groups on the low-channel converted image 12. Further, for example, information input to the second learning model 20 may be classified and input by channels or color groups constituting the low-channel converted image 12.

In this case, as described above, the plurality of color groups are, for example, a first color group composed of a first signal color corresponding to the color of a first signal of a traffic light and a second signal color composed of a second signal color corresponding to the color of a second signal of a traffic light. It can be composed of two color groups.

In this case, for example, the first color group may be a color group having a color corresponding to red, and the second color group may be a color group having a color corresponding to green.

In addition, it is assumed that a traffic light is composed of only the first and second signals of red and green, but the plurality of color groups are only the first color group and the second color group above in case the traffic light also includes a third signal of yellow color. In addition, it should be noted that an additional color group corresponding to the third signal may be included.

In the candidate region selection step S2, geometric information about a plurality of color groups for input to the second learning model 20 may be first extracted. At this time, since each color group has one or more elements of position or size differently formed, information on the position and size of each color group above may be used as information for distinguishing each color group.

At this time, the location information of each color group may be based on, for example, a center point of the color group. In addition, the location information of each color group is determined based on the square passing through the coordinates by selecting, for example, the x-coordinates of the pixels on the left and right of each color group, and selecting the y-coordinates of the pixels on the top and bottom sides. It may be.

On the other hand, it is okay to set the location information of the color groups in a different manner from the above examples.

In this case, the information on the center point of the color group may be extracted as coordinate information on a coordinate plane set on the low-channel transformed image 12, for example. Also, for example, the size information of each color group may be the size of the horizontal and vertical length of the color group.

The geometric information of the plurality of color groups may be separately extracted before information on the plurality of color groups is input to the second learning model 20, but the low-channel converted image 12 is directly applied to the second learning model 20. It is also possible to input and extract geometric information of each color group directly from the low-channel converted image 12 in the second learning model 20.

The extracted geometric information of each color group is input to the second learning model 20. Meanwhile, the geometric information of each color group may undergo a normalization process before being input to the second learning model 20. This is to reduce the possibility of an error occurring according to the deviation of the screen size.

The second learning model 20 determines whether each color group corresponds to an actual traffic light color sphere based on the data obtained by the already performed learning. To this end, the second learning model 20 calculates the probability that each color group corresponds to an actual traffic light color sphere.

At this time, for example, the corresponding probability may be calculated based on whether the position and size information of each color group on the low-channel converted image 12 corresponds to the position and size information of an actual traffic light color sphere.

On the other hand, since traffic lights are generally located at a certain height apart from the road, there is no significant change in their location. In addition, in the shape of a traffic light, three or four color spheres are generally arranged in a rectangular frame, and the size is mostly standardized, so there is no significant difference.

Therefore, it can be seen that such a judgment process has particularly high utility and accuracy because the location, shape, and size of traffic lights on an actual road are somewhat limited.

Through this process, a traffic light correspondence probability, which is a probability that each color group corresponds to a traffic light color sphere, is calculated, and a color group with a traffic light correspondence probability lower than a certain probability may be determined as meaningless data.

That is, since a color group having a traffic light response probability lower than a certain probability can be certain that it is not a traffic light, the corresponding color groups are not used as data for determining the state of the traffic light in the traffic light state determination step S3.

At this time, the reference probability, which is the criterion for determination, can be arbitrarily set to a value having an appropriate reliability. In addition, a color group having a traffic light response probability higher than the reference probability may be tentatively determined as a color group corresponding to the traffic light color sphere.

In addition, an area on the input image 11 corresponding to an area in which a color group determined to be represented by a traffic light color sphere on the low-channel converted image 12 may be designated as the traffic light candidate area 17. At this time, the traffic light candidate region 17 refers to an area determined to correspond to a traffic light on the input image 11.

10 is a flow chart illustrating a learning process of the second learning model 20 in an embodiment of the present invention. Hereinafter, a learning process of the second learning model 20 in an embodiment of the present invention will be described in detail with reference to FIG. 10.

Like the first learning model 10, the second learning model 20 may be learned by a supervised learning method. In addition, the second learning model 20 may be composed of a CNN similar to the first learning model 10. At this time, in order to distinguish the CNN constituting the second learning model 20 from the CNN constituting the first learning model 10, it will be referred to as a second CNN.

In order to learn the second learning model 20, first, a plurality of low-channel transformed images 12 including one or more color groups are secured. At this time, at least one color group is an area in which the same color is clustered on the low-channel converted image 12 converted through the image conversion step S1 as described above.

Thereafter, training data indicating whether the plurality of color groups correspond to the actual traffic light color sphere is created and collected.

In this case, the above display information may be information obtained by directly selecting and displaying a color group corresponding to an actual traffic light color sphere among a plurality of color groups existing on the low-channel converted image 12. Meanwhile, the set of training data secured in this way will be referred to as second training data.

In addition, a plurality of probability data output in the candidate region selection step (S2) is secured.

As described above, the probability data is calculated based on geometric information for a plurality of color groups on the low-channel transformed image 12 in the candidate region selection step (S2), as described above, and contains information on the probability that each color group corresponds to a color sphere of a traffic light. Means set.

Thereafter, supervised learning is performed based on the probability data and the second training data. At this time, supervised learning may be performed by going through a process of checking whether each probability information constituting the probability data matches or does not match the information of the second training data.

By performing such supervised learning, error optimization can be performed. That is, as the number of times supervised learning is performed increases, the probability of an error may decrease. As a result, as the learning experience of the second learning model 20 increases, the traffic light candidate region 17 selected in the candidate region selection step S2 may more accurately correspond to the actual traffic light position.

11 is a flow chart showing a detailed process of the traffic light state determination step (S3) according to an embodiment of the present invention, and FIG. 12 is a traffic light of the traffic light candidate region 17 in the traffic light state determination step (S3) according to an embodiment of the present invention. It represents the state in which the state is detected.

Hereinafter, a detailed process of determining a traffic light state (S3) according to an embodiment of the present invention will be described with reference to FIGS. 11 and 12.

First, the traffic light candidate region 17 selected in the candidate region selection step (S2) is input into the third learning model 30.

The third learning model 30 determines the state of the traffic light by analyzing the information on the inputted traffic light candidate region 17. At this time, the determined traffic light state means signal state and location information in the image, and thus, the color state of the color sphere of the traffic light can be determined. That is, it may be determined whether the signal displayed on the color sphere of the traffic light is red or green.

In addition, the determined traffic light state may include recognition of a mark displayed on the color sphere of the traffic light. At this time, the above sign may be an indication sign for the direction of the vehicle, such as a left turn or a right turn.

Meanwhile, in this case, the value first output in the step S3 of determining the traffic light state may be a probability value of the existence of each corresponding traffic light state. That is, in the traffic light state determination step S3, a probability that the input signal light candidate region 17 is in a specific signal state may be calculated and a corresponding probability value may be output.

In this case, if the output probability value has a value higher than the preset reference probability, it may be determined that the signal displayed on the corresponding traffic light candidate region 17 corresponds to a signal marked with a specific color or mark.

In addition, the output of the third learning model 30 may further include a probability of the presence of the second traffic light. In this way, by calculating the probability of the existence of a traffic light in the traffic light state determination step (S3), it is possible to check whether the corresponding traffic light candidate region 17 actually corresponds to an area corresponding to the traffic light after the candidate region selection step (S2). .

At this time, among the plurality of traffic light candidate regions 17, the traffic light candidate region 17, which is found not to actually correspond to a traffic light, may be excluded in determining the state of the traffic light.

Through this process, as a result, the traffic light state result data 18 can be derived as shown in FIG. 12. In FIG. 12, the traffic light state result data 18 is a state in which it is determined that the green signal is lit in the actual traffic light color sphere from the analysis of the detected traffic light candidate region 17.

13 is a flow chart showing a learning process of the third learning model 30 in the traffic light detection method according to an embodiment of the present invention. Hereinafter, the learning process of the third learning model 30 will be described in detail with reference to FIG. 13.

Like the first learning model 10 and the second learning model 20, the third learning model 30 may be learned by a supervised learning method.

In order to learn the third learning model 30, first, data including a plurality of traffic light candidate regions 17 are secured. In this case, the above data may be data related to the input image 11 including a plurality of traffic light candidate regions 17.

In addition, the traffic light candidate regions 17 may be traffic light candidate regions 17 selected based on geometric information about a plurality of low-channel transformed images 12 in the candidate region selection step (S2) according to an embodiment of the present invention. have.

Thereafter, based on the plurality of upper traffic light candidate regions 17, training data displayed for the position and state information of each of the traffic light candidate regions 17 are accumulated.

At this time, the display of the location and status information may be directly displayed by a person, but if a method capable of collecting data that can serve as a correct answer to the third learning model 30, the above data is accumulated by another method. It is also okay.

A set of training data that displays the positions and states of the plurality of traffic light candidate regions 17 secured in this manner will be referred to as third training data.

In addition, a plurality of traffic light status information is secured. At this time, the plurality of traffic light status information refers to a set of information on the status of a traffic light output by performing the traffic light status determination step (S3) in an embodiment of the present invention. This can be used to mean including the traffic light condition result data described above.

Thereafter, the third learning model 30 is learned based on the obtained plurality of traffic light state information and the third training data. Specifically, based on the third training data, the third learning model 30 may be trained by repeatedly determining whether or not the plurality of traffic light state information is correct information.

In addition, since the third learning model 30 repeats learning, the possibility of making an erroneous determination in the traffic light state determination step (S3) gradually decreases, so error optimization can be performed by repeating learning.

The above description is merely illustrative of the technical idea of the present embodiment, and those of ordinary skill in the technical field to which the present embodiment belongs will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present exemplary embodiments are not intended to limit the technical idea of the present exemplary embodiment, but are illustrative, and the scope of the technical idea of the present exemplary embodiment is not limited by these exemplary embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

10: first learning model 20: second learning model
11: Input image 30: Third learning model
12: Low-channel converted video
13: 1st channel
14: second channel
15: 3rd channel
16: color gamut response area
17: traffic light candidate area
18: traffic light condition result data

Claims (13)

An image conversion step of converting, by the first learning model, an input image photographing a traffic light and its surrounding environment into a low-channel converted image composed of at least a first signal color, a second signal color, and a background color;
A candidate region in which the second learning model selects one or more traffic light candidate regions based on a plurality of first color groups having the first signal color and a plurality of second color groups having the second signal color in the low-channel converted image Screening step;
A traffic light state determination step in which a third learning model determines the state of the traffic light based on partial images of the input image corresponding to the traffic light candidate region.
Traffic light detection method comprising a. The method of claim 1,
And in the selecting of the candidate region, the traffic light candidate region is selected based on a probability that the first color group and the second color group correspond to the position of the traffic light color sphere on the traffic light. The method of claim 1,
The traffic light detection method, characterized in that the low-channel converted image further includes a third signal color. The method of claim 1,
The first learning model comprises a first training data that accumulates information about the traffic light based on a plurality of images and a first CNN that performs supervised learning based on the plurality of low-channel transformed images. The method of claim 4,
The first learning model outputs a first error value, which is a difference value between the information on the traffic light region among the first training data and the information on the region corresponding to the traffic light region among the plurality of low-channel converted images. A traffic light detection method, characterized in that error optimization is performed in which a weight is assigned only to the first error value among error values. The method of claim 4,
The traffic light detection method, characterized in that the information on the traffic light is position information of the traffic light displayed by a person. The method of claim 1,
The second learning model includes a second training data created based on a plurality of the plurality of color groups, and a second CNN that performs supervised learning based on a probability of matching between the plurality of color groups and the traffic light color area of the traffic light. Way. The method of claim 7,
And the second training data is training data indicating whether or not the plurality of color groups match the color sphere of the traffic light. The method of claim 1,
Wherein the third learning model performs supervised learning based on third training data created based on the traffic light candidate region and the determined state information of the traffic light. The method of claim 9,
And the third training data is training data in which a person displays location information and status of the traffic light based on the traffic light candidate area. The method of claim 10,
The traffic light detection method, characterized in that the location information of the traffic light is a center, a horizontal length, and a vertical length of the traffic light. The method of claim 10,
The traffic light detection method, characterized in that the status information of the traffic light comprises a color or direction indication of a color sphere of the traffic light. A traffic light detection device that performs traffic light detection by the traffic light detection method according to claim 1.

Images