KR20210097782A - Indicator light detection method, apparatus, device and computer-readable recording medium - Google Patents

Abstract

The present invention relates to a method, an apparatus, a device, and a computer-readable recording medium for detecting an indicator light. The method includes: recognizing the collected road images to obtain candidate bounding boxes of indicator lights in the road images; predicting a plurality of classifications of indicator lights based on an image region corresponding to the candidate bounding box in the road image to obtain a plurality of classification prediction results of the indicator lights; and determining the display state of the indicator light based on a plurality of classification prediction results of the indicator light, wherein the plurality of classifications include at least two of a use classification, a shape classification, an arrangement classification, a function classification, a color classification, and an orientation classification. includes

Description

Indicator light detection method, apparatus, device and computer-readable recording medium

[Citation of related applications]

The present invention claims the priority of a Chinese patent application whose filing date is July 31, 2019, the application number is CN2019107037635, and the title of the invention is "Indicator detection method, apparatus, device and computer readable recording medium", All contents of the Chinese patent application are incorporated herein by reference.

[Technology]

The present invention relates to the field of computer vision technology, and more particularly, to a method, an apparatus, a device, and a computer-readable recording medium for detecting an indicator light.

The detection of indicator lights is a very important part in autonomous driving of vehicles and autonomous driving of robots. In the process of autonomous driving of a vehicle or robot, in order to make a judgment that meets traffic rules for safe driving, it is necessary to detect an indicator light from a road image captured by a camera and determine the state and meaning of the indicator light. .

In order to overcome the problems existing in the related art, the present invention provides a method, an apparatus, a device, and a computer-readable recording medium for detecting an indicator light.

According to a first aspect, there is provided a method for detecting an indicator light, the method comprising: recognizing a collected road image to obtain a candidate bounding box of an indicator light in the road image; predicting a plurality of classifications of indicator lights based on an image region corresponding to the candidate bounding box in the road image to obtain a plurality of classification prediction results of the indicator lights; and determining the display state of the indicator light based on a plurality of classification prediction results of the indicator light, wherein the plurality of classifications include at least two of a use classification, a shape classification, an arrangement classification, a function classification, a color classification, and an orientation classification. includes

According to a second aspect, there is provided an indicator light detecting apparatus, comprising: a recognition unit for recognizing a collected road image to obtain a candidate bounding box of an indicator light in the road image; a prediction unit for predicting a plurality of classifications of indicator lights based on an image region corresponding to the candidate bounding box in the road image to obtain a plurality of classification prediction results of the indicator lights; and a determining unit configured to determine a display state of the indicator light based on a plurality of classification prediction results of the indicator light, wherein the plurality of classifications include one of usage classification, shape classification, arrangement classification, function classification, color classification, and orientation classification. include at least two

According to a third aspect, there is provided an indicator light detection device, the device comprising a memory and a processor, the memory storing computer instructions executable on the processor, the processor when the computer instructions are executed, the method described above to realize

According to a fourth aspect, there is provided a computer-readable recording medium having a computer program stored thereon, wherein the method described above is realized when the program is executed by a processor.

According to a fifth aspect, there is provided a computer program comprising computer readable code,

The method described in the embodiment of the present invention is realized when the computer readable code is executed by a computer.

The indicator light detection method, apparatus, device and computer readable recording medium of one or more embodiments of the present invention can detect and classify the indicator light using a road image collected by a camera, without having to rely on a high-precision sensor or the like. By implementing it, it is possible to effectively reduce the cost of device hardware required for detecting the indicator light. In the process of recognizing the image region corresponding to the candidate bounding box of the indicator light in the road image, by executing a clear classification logic division on the indicator light, the indicator light is classified into a plurality of aspects and a plurality of dimensions, and a plurality of classification predictions obtained Since the result covers the indicator light in each case as much as possible, and it is advantageous for the judgment of the display state of the indicator light in various situations, it is possible to effectively improve the comprehensiveness and accuracy of the indicator light detection.

It should be understood that the foregoing general description and the following detailed description are illustrative and explanatory only, and not limitation to the present invention.

The drawings herein are incorporated in and constitute a part of this specification. These drawings show embodiments consistent with the present invention, and together with the specification may be used to describe embodiments of the present invention.
1 is a schematic diagram showing a flow of a method for detecting an indicator light according to an exemplary embodiment of the present invention.
Fig. 2 is a schematic diagram showing the indicator light classification logic according to an exemplary embodiment of the present invention.
3A is a schematic diagram showing the configuration of a neural network model according to an exemplary embodiment of the present invention.
Fig. 3B is a schematic diagram showing a flow of a method in which the neural network model in Fig. 3A executes a plurality of classification predictions for indicators.
4 is a schematic diagram showing a result of detecting an indicator light according to an exemplary embodiment of the present invention.
5 is a schematic diagram illustrating a flow of a method for determining whether a detected indicator light is the same location indicator light according to an exemplary embodiment of the present invention.
6 is a schematic diagram showing a flow of a method for training a neural network model according to an exemplary embodiment of the present invention.
Fig. 7 is a schematic diagram showing the configuration of an indicator light detecting apparatus according to an exemplary embodiment of the present invention.
8 is a block diagram showing an indicator light detecting device according to an exemplary embodiment of the present invention.

Exemplary embodiments are described in detail herein, examples of which are shown in the drawings. When the following description refers to drawings, the same reference numerals in different drawings refer to the same or similar elements, unless otherwise specified. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. To the contrary, these are merely examples of apparatus and methods consistent with several aspects of the invention as detailed in the appended claims.

As used herein, the term “and/or” merely describes the related relationship of the related subject, for example, A and/or B are: A alone, A and B concurrently, and It means that there may be three relationships, such as the existence of alone B. The term "at least two" in this specification also means any two of a plurality of species, or any combination of at least two of a plurality of species. That is, for example, including at least one of A, B, and C means any two or a plurality of elements selected from the group consisting of A, B and C.

1 is a schematic diagram showing one flow of a method for detecting an indicator light according to an embodiment of the present invention. As shown in Fig. 1, the method of this embodiment includes steps 110 to 130.

In step 110, the collected road images are recognized to obtain candidate bounding boxes of indicator lights in the road images.

In the driving process of an intelligent device such as a vehicle or a robot, road images around the intelligent device are collected by using the intelligent device or at least one image collecting device (eg, a camera, etc.) installed around the intelligent device.

Through recognition of the collected road images, for example, by inputting the collected road images into a pre-trained neural network model, candidate bounding boxes for indicators in the road images can be obtained. The indicator light includes, for example, a traffic signal light, a railroad signal, etc., and the present invention is not limited to the type of the indicator light.

In step 120, based on the image region corresponding to the candidate bounding box in the road image, a plurality of classifications of indicator lights are predicted to obtain a plurality of classification prediction results of the indicator lights.

Here, the plurality of classifications include at least two of a use classification, a shape classification, an arrangement classification, a function classification, a color classification, and an orientation classification.

The plurality of classification refers to classification of the indicator light into a plurality of aspects and a plurality of dimensions. By designing the indicator light classification logic, the classification of the indicator light can cover various types of indicator lights.

In the embodiment of the present invention, it is necessary to predict at least two classifications among the plurality of classifications, and to perform type prediction for each sub-classification of the two classifications.

In step 130, the display state of the indicator lamp is determined based on prediction results of a plurality of classifications of the indicator lamp.

It is possible to determine different display states of the indicator lights according to different situations of the prediction results of each classification.

In an embodiment of the present invention, the display state of the indicator light is determined based on the prediction result of each of the at least two classifications.

According to this embodiment, it is possible to effectively reduce the cost of device hardware required for detecting the indicator light by executing the detection and classification of the indicator light using the road image collected by the camera, without having to rely on a high-precision sensor or the like. . In the process of recognizing the image region corresponding to the candidate bounding box of the indicator light in the road image, by executing a clear classification logic division on the indicator light, the indicator light is classified into a plurality of aspects and a plurality of dimensions, and a plurality of classification predictions obtained Since the result covers the indicator lights of various situations as much as possible, and is advantageous in judging the display state of the indicator lights of various situations, it is possible to effectively improve the comprehensiveness and accuracy of the indicator lamp detection.

In the following description, the indicator light detection method will be described in more detail.

2 illustrates an exemplary indicator light classification logic. As shown in FIG. 2 , the usage classification may include, for example, that the indicator light is used to indicate a pedestrian / that the indicator light is used to indicate a vehicle. When the indicator light is used to indicate a vehicle, the shape classification may include, for example, that the indicator light belongs to a full screen light (also called a circular light) / arrow or the like, and the arrangement classification is, for example, It may include that the indicator light belongs to a horizontal arrangement / belongs to a vertical arrangement / belongs to a single indicator light, and the function classification is, for example, that the indicator light belongs to a normal indicator / warning, etc. / electronic toll collection (Electronic Toll Collection, ETC) may include those belonging to the indicator light, and the color classification is, for example, that the indicator belongs to red, etc. / belongs to yellow, etc. / belongs to green, etc. / unknown color corresponding) may be included. When the indicator light is an arrow or the like, the direction classification may include, for example, left/right/front/left front/right front. Those skilled in the art should understand that the classification of the indicator light is not limited to the above, and may include classification of other aspects or dimensions.

In order to obtain a plurality of classification prediction results of indicators, a neural network model may be trained (training process will be described in detail later) using a road image (referred to as a sample image) including previously labeled information, and The obtained neural network model recognizes an indicator in the input road image, predicts a plurality of classifications of the recognized indicator, and obtains a plurality of classification prediction results.

In some embodiments, a plurality of classifications of the indicator light may be predicted by using a plurality of subnetwork branches included in the neural network model, and a plurality of classification prediction results of the indicator light may be obtained, where the plurality of The number of subnetwork branches is equal to the plurality of classification quantities, and each subnetwork branch is used to recognize a subtype of one of the plurality of classifications.

3A is a schematic diagram of a network configuration of a neural network model provided by at least one embodiment of the present invention. As shown in FIG. 3A , the neural network model includes a feature extraction layer 301 and a region candidate (Region Proposal Network). ) layer 302 , a pulling layer 303 and a fully connected layer 304 . Here, the fully connected layer 304 includes a regression branch 3041 and a plurality of subnetwork branches 3042 .

In one optional embodiment, the fully connected layer 304 further comprises a convolutional layer, the convolutional layer being connected with the pooling layer 303 .

Fig. 3B is a schematic diagram showing a flow of a method for executing a plurality of classification predictions of indicators using the neural network model in Fig. 3A, and as shown in Fig. 3B, the method includes steps 310 to 340;

In step 310, a feature map of the road image is obtained using the feature extraction layer.

The feature extraction layer 301 is used to extract features of the input road image, and the feature extraction layer 301 may be a convolutional neural network, for example, an existing Visual Geometry Group (Visual Geometry Group). , VGG) networks, residual networks (ResNet), dense connection networks (DenseNet), etc. may be employed, and other convolutional neural networks may also be employed. The present invention does not limit the specific configuration of the feature extraction layer 301 . In an optional embodiment, the feature extraction layer 301 may include network units such as a convolution layer, an activation layer, a pooling layer, and the like, and the above-described network units may be stacked in a certain manner. Here, the convolution layer can obtain a plurality of feature maps by extracting different features from each input road image using a plurality of convolution kernels, and the pooling layer is located behind the convolution layer, Local averaging and downsampling are performed, and the resolution of the feature map can be reduced. As the number of convolutional layers and pooling layers increases, the number of feature maps gradually increases, and the resolution of the feature maps gradually deteriorates.

In step 320, processing is performed on the feature map by using the area candidate layer to generate a candidate bounding box of an indicator light in the road image.

The region candidate layer 302 is used to predict a candidate bounding box of an indicator, that is, generates prediction information of a candidate bounding box. The region candidate layer 302 may be in a region candidate network (RPN), and the present invention does not limit the specific configuration of the region candidate layer 302 . In an optional embodiment, the region candidate layer 302 may include network units such as a convolution layer, a classification layer, a regression layer, and the like, and the above-described network units are generated by stacking them in a certain manner. Here, the convolution layer performs convolution on the input feature map using a sliding window (for example, 3*3), each window corresponds to a plurality of anchor boxes, and each window has one Generate vectors for full access to the classification and regression layers of The classification layer determines whether the image region in the candidate bounding box generated by the anchor box is foreground or background, and the regression layer is used to obtain the approximate position of the candidate bounding box, and is based on the output results of the classification layer and the regression layer. Based on the prediction, the candidate bounding box including the indicator light is predicted, and the probability that the image area in the candidate bounding box is the foreground or the background and the position parameter of the candidate bounding box are output.

In this step, the generated candidate bounding box may be called a Region Proposal, and in a subsequent step, the corresponding region may be called a Region of Interest (ROI).

In step 330, an image feature of a corresponding predetermined size in the feature map of the candidate bounding box is obtained by using the pooling layer.

In this step, the pooling layer 303 is employed to extract a feature map of a predetermined size (a fixed size, for example, 7Х7) for each ROI, that is, ROIs having different sizes obtained in step 320 are sized. maps to the same region, and this process can be called ROI polling. By employing the pooling layer 303 to perform feature extraction on regions having the same size, image features corresponding to the ROI can be obtained.

In step 340 , a plurality of classification prediction results of the indicator light are obtained through the fully connected layer 304 . The fully connected layer 304 includes a regression branch 3041 and a plurality of subnetwork branches 3042 . The regression branch 3041 and each subnetwork branch 3042 further perform feature extraction using a convolution kernel of 1Х1, respectively. The convolutional kernel of 1Х1 has different parameters for each channel, and corresponds to a function of complete connection.

Here, the regression branch 3041 performs regression on the above-described candidate bounding box, correcting the position of the candidate bounding box, thereby more accurately positioning the bounding box of the indicator light. The regression branch 3041 is obtained by predicting the bounding box of the indicator using the transformation relationship between the candidate bounding box and the actual bounding box obtained through learning in the training process, that is, it can be obtained by predicting the position information of the bounding box of the indicator. there is. The location information may be expressed as (x1, y1, x2, y2), where x1 and y1 are the coordinates of the upper left corner of the predicted bounding box, and x2, y2 are the coordinates of the lower right corner of the predicted bounding box. The location information may be expressed as (x, y, w, h), where x and y represent the coordinates of the predicted bounding box center point, and w and h represent the predicted width and height of the bounding box, respectively.

Here, each subnetwork branch 3042 is used to recognize a subtype of one classification among the plurality of classifications described above.

In an example, the subnetwork branch may obtain a plurality of classification prediction results such as an indicator by the following method, wherein the method includes an image feature corresponding to the candidate bounding box and a first among the plurality of subnetwork branches. predicting a first classification among a plurality of classifications of the indicator light by using one subnetwork branch to obtain prediction probabilities of at least two subtypes corresponding to the first classification; and marking a subtype having the highest prediction probability among the at least two subtypes as a subtype of the first classification of the indicator light.

Here, the first subnetwork branch may be any one of a plurality of subnetwork branches, and the first classification may be any one of a plurality of classifications.

For example, the first subnetwork branch may be a subnetwork branch for performing shape classification, and prediction probabilities of two subtypes (full screen light and arrow, etc.) corresponding to shape classification may be obtained. For example, the prediction probability of a full screen light is 90%, and the prediction probability of an arrow etc. is 10%. In this case, the sub-network branch in question marks the subtype with the highest prediction probability, that is, full screen light, as the subtype of shape classification.

Since there is a hierarchical bar, that is, a logical relationship between the plurality of classifications of the indicator lights, it is necessary to jointly determine the display state of the indicator lights based on the prediction results of the plurality of classifications of the indicator lights.

In some embodiments, the display state of the indicator light is determined using the following method.

When the plurality of classifications include the use classification and the shape classification, and the prediction result of the use classification is that the indicator light indicates a vehicle, and the prediction result of the shape classification is that the indicator light is a circular light, the arrangement a first display state of the indicator light is obtained by combining the prediction results corresponding to the classification, the function classification, and the color classification, respectively;

When the plurality of classifications include the use classification and the shape classification, and the prediction result of the use classification is that the indicator indicates a vehicle, and the predictive result of the shape classification is that the indicator light is an arrow, the color combining the classification and the prediction result respectively corresponding to the directed classification to obtain a second display state of the indicator light,

When the plurality of classifications include the use classification and the prediction result of the use classification is that the indicator light indicates a pedestrian, a third display state of the indicator light is obtained.

Fig. 4 shows an exemplary output result of indicator light detection, as shown in Fig. 4 , the detection result includes a bounding box of an indicator light obtained by prediction and a plurality of classification prediction results of the indicator light, and a boundary obtained by prediction It may further include evaluating the reliability of the box. The reliability evaluation comprehensively reflects the possibility that the indicator light is present in the bounding box obtained by the prediction and the accuracy of the position of the bounding box obtained by the prediction.

As shown in Fig. 4, in the road image (only a part of the road image is displayed to clearly display the prediction result to be output), a bounding box of three indicator lights obtained by prediction is output. Here, the classification prediction result of the indicator in the black rectangular bounding box is “red horizon”, indicating that the indicator is horizontally arranged and is a red normal indicator, and the bounding box obtained by the prediction is The reliability evaluation is 1.0. Since the classification prediction result of a normal indicator light is set to not be displayed (not displayed), the display state of the indicator lamp corresponds to the first display state.

The classification prediction result of the indicator in the black square bounding box is "unknown color, single", that is, the indicator is independent and indicates that the indicator is not lit, and the prediction results of these two classifications are set to not displayed. Therefore, it is not displayed in the image and corresponds to the first display state.The reliability evaluation of the bounding box obtained by the prediction is 0.98.

The classification prediction result of the indicator in the white square is "green, arrow, left", the indicator is a single arrow-shaped indicator, the color is green, and the direction is left, and the result obtained by the prediction The reliability evaluation of the bounding box is 0.99 The display state of the indicator light corresponds to the second display state.

Determining the display state of the indicator light further includes determining whether the indicator light is always on or flashing, so that the intelligent device can better guide the driving automatically, and the intelligent device complies with the traffic rules to drive safely to realize

In some embodiments, the indicator light and a plurality of classification prediction results of the indicator light at the same position in the plurality of road images collected within a predetermined time are obtained, and a plurality of classifications of the indicator light at the same position in the plurality of road images are obtained. The display state of the indicator light is determined based on the prediction result.

In one optional embodiment, it is possible to collect consecutive frames of road images, and obtain the indicator lights at the same location in the plurality of road images. The image of the continuous frame may be an image of a plurality of frames taken continuously, and a plurality of consecutively selected frames by selecting one target frame at regular intervals from several frames from among the images of a plurality of frames taken continuously The target frame of may be regarded as a continuous frame.

Fig. 5 is a schematic diagram showing a flow of a method for determining whether a detected indicator light is an indicator light at the same position, and as shown in Fig. 5 , the method includes the following steps.

In step 510, the position of the indicator light in the image of the first frame of the road image of the successive frame is obtained.

That is, the position of the indicator lamp obtained by prediction within the start frame within a predetermined time is obtained.

According to the indicator light detection method described above, after detecting the indicator lamp in the road image, the position of the indicator lamp in the image can be obtained using the neural network model, that is, the position parameter of the bounding box of the indicator lamp obtained by prediction can be obtained. Also, according to the manner of image processing, the position of the detected indicator light in the image can be obtained.

In step 520, based on the position of the indicator light in the image of the first frame and the movement speed and shooting frequency of the device for capturing the road image, the first frame of the road image of the consecutive frame of the indicator light is A first position in the image of each frame other than the image is calculated.

The device for photographing the road image is an image collection device (eg, a camera, etc.), the movement speed of the device is the same as the movement speed of the automatically running intelligent device, and the photographing frequency is preset or the configuration of the device is read is obtained by If the position of the indicator in the image in the start frame is already known, the indicator in each subsequent frame (that is, an image other than the first frame in the images of other frames in successive frames) based on the movement speed and shooting frequency of the device. We can compute the theoretical position within the image of By calling this position the first position, it is distinguished from the position of a subsequent step.

In step 530, a second position in the image of each other frame of the image of the successive frame of the indicator light is obtained.

For each subsequent frame, the position in the image of the detected indicator light can be predicted using the neural network model, or the position of the detected indicator light can be obtained through the image processing method, and the position is referred to as the second position. call

In step 540, if the difference between the second position and the first position is less than a predetermined value for the image of each frame among the images of the other frames, the indicator lights detected in the road images of the successive frames are the same Confirm that it is a position indicator light.

For the indicator light at the same position, the second position in the indicator light image detected in step 530 may be close to the first position calculated in step 520 . Therefore, for each subsequent frame, if the difference between the second position and the first position is less than a predetermined value, it is determined that the indicator light detected in the road image of the successive frames is the indicator light of the same position, otherwise It is judged that the indicator lights are not in the same position. If the detected indicator lights are not indicators of the same position, there is no need to execute the step of determining the status of the subsequent indicator lights. A person skilled in the art should understand that the above-described predetermined value can be set according to the required detection accuracy.

After obtaining the indicator light at the same position in a plurality of road images, by comparing whether a plurality of classification prediction results of the indicator light at the same position in a plurality of road images are maintained or changed, whether the indicator light is always on or blinks state can be determined.

In one optional embodiment, when the color classification prediction results of the indicator lights at the same location in the plurality of road images are the same, it is determined that the display state of the indicator lights is always on,

When the color classification prediction result of the indicator lamp at the same position in the plurality of images changes at intervals, it is determined that the display state of the indicator lamp is blinking.

For example, if the prediction result of color classification of the indicator light at the same position in the plurality of road images described above is the same, it means that the color of the indicator light does not change within a predetermined time (for example, 3 seconds). , it may be determined that the indicator light is always on. It is worth noting that color classifications with the same prediction result here do not include unknown colors.

If the prediction result of color classification of the indicator light at the same position in the above-described plurality of road images changes at intervals, for example, the prediction result of color classification within a certain period is green, and the color classification of the color classification within another period of time is green. If the predicted result is an unknown color (or it cannot be detected that there is a light here), and these two situations alternate, it means that the color of the indicator light has alternately changed within a predetermined time. , it may be determined that the indicator is in a blinking state.

The following description describes how to train a neural network model. 6 is a schematic diagram showing one flow of a training method for a neural network model according to an embodiment of the present invention. As shown in FIG. 6 , the method of this example includes the following steps.

In step 610, a sample image including indicator lights is input to the neural network model, and a plurality of classification prediction results of indicator lights and bounding box prediction results are obtained.

Before training, initialization is performed on the neural network model to determine the initialized network parameters.

The sample image input to the neural network model may be a road image including an indicator, and labeling information of the indicator may be pre-labeled in the sample image, and the labeling information includes actual bounding box information of the indicator, For example, it includes the coordinates of the upper left vertex and the coordinates of the lower left vertex of the bounding box, and the labeling information further includes a plurality of classification information such as an indicator.

By inputting the sample image to an initialized neural network model, a plurality of classification prediction results and bounding box prediction results of indicators in the sample image may be predicted and obtained.

In step 620, a loss value of the loss function is calculated based on the plurality of classification prediction results, the boundary box prediction results, and the plurality of classification information and the actual boundary box information.

The loss value of the loss function represents a difference between a plurality of classification results obtained by prediction and a bounding box obtained by prediction, and a plurality of classification information and actual bounding box information that have been labeled in advance.

In step 630, a network parameter of the neural network model is adjusted based on the loss value.

In an optional embodiment, the network parameter is adjusted by sending a loss value determined based on the loss function to the neural network model, for example, the value of the convolution kernel of each layer, and the weight of each layer. Adjust parameters, etc.

When training a neural network model, the training sample is divided into a plurality of image subsets (batch), and one image subset is sequentially input to the neural network model in each repetition training, The loss value of the prediction result of each sample in the training sample is combined to perform adjustment of the network parameter. After this iterative training is completed, the next one iteration training is executed by inputting the next one image subset to the neural network model. The training samples included in the different image subsets differ at least in part. When a predetermined termination condition is reached, training of the neural network model may be completed. The predetermined training termination condition may include, for example, that a loss value has decreased to a certain threshold value, or that a predetermined number of iterations of the neural network model has been reached.

According to the training method of the neural network model of this embodiment, by training the neural network model using the sample image in which the classification information of the indicator and the actual bounding box are pre-labeled, the indicator in the image input by the trained neural network model is detected. and a plurality of classifications of the indicator light can be predicted.

The trained neural network model is a neural network model employed in the above-described embodiment of the indicator light detection method, and its configuration is, for example, as shown in FIG. 3A, and if it is a distinguishing point, only the input image is a sample containing labeling information It's just an image. In the case of the neural network model shown in FIG. 3A , obtaining the prediction result of the indicator light based on the sample image includes: obtaining a feature map of the sample image using the feature extraction layer; performing processing on the feature map using the region candidate layer and generating a candidate bounding box of an indicator light in the sample image; obtaining an image feature of a corresponding predetermined size from the feature map of the candidate bounding box by using the pooling layer; and obtaining a plurality of classification prediction results of the indicator light and a prediction result of a bounding box by using the fully connected layer.

Since the indicator light prediction process in the training process is similar to the indicator light prediction process in the indicator light detection method, the detailed process may refer to the description of the embodiment of the indicator light detection method.

7 provides an indicator light detecting device, as shown in FIG. 7 , the device may include a recognizing unit 701 , a predicting unit 702 , and a determining unit 703 .

Here, the recognition unit 701 is used to recognize the collected road image to obtain a candidate bounding box of an indicator light in the road image, and the prediction unit 702 is an image area corresponding to the candidate bounding box in the road image. is used to predict a plurality of classifications of the indicator lights to obtain a plurality of classification prediction results of the indicator lights, and the determining unit 703 determines the display state of the indicator lights based on the plurality of classification prediction results of the indicator lights. used to determine, wherein the plurality of classifications include at least two of a use classification, a shape classification, an arrangement classification, a function classification, a color classification, and an orientation classification.

In another embodiment, the prediction unit 702 performs feature extraction on an image region corresponding to the candidate bounding box by using a neural network model to obtain an image feature corresponding to the candidate bounding box, and further A plurality of classifications of the indicator are predicted by using an image feature corresponding to a candidate bounding box and a plurality of subnetwork branches included in the neural network model to obtain a plurality of classification prediction results of the indicator, where the plurality of classifications are predicted. The number of sub-network branches of A is equal to the quantity of the plurality of classifications, and each subnetwork branch is used to recognize a subtype of one of the plurality of classifications.

In another embodiment, the prediction unit 702 is configured to use an image feature corresponding to the candidate bounding box and a first subnetwork branch among the plurality of subnetwork branches to determine a first classification among a plurality of classifications of the indicator light. Prediction to obtain prediction probabilities of at least two subtypes corresponding to the first classification. A subtype having the highest prediction probability among the at least two subtypes is marked as a subtype of the first classification of the indicator light.

In another embodiment, the determining unit 703 is configured such that the plurality of classifications include the use classification and the shape classification, and the prediction result of the use classification is that the indicator light indicates a vehicle, and the shape classification when the prediction result of indicates that the indicator light is a circular light, the first display state of the indicator light is obtained by combining the prediction results respectively corresponding to the arrangement classification, the function classification and the color classification, or the plurality of classifications are the At the same time including the usage classification and the shape classification, when the prediction result of the usage classification is that the indicator indicates a vehicle, and the prediction result of the shape classification is that the indicator light is an arrow, etc., the color classification and the orientation classification The second display state of the indicator light is obtained by combining the corresponding prediction results, and when the plurality of classifications include the use classification, and the prediction result of the use classification is that the indicator light indicates a pedestrian, A third display state of the indicator light is obtained.

In another embodiment, the determining unit 703 obtains the indicator light and a plurality of classification prediction results of the indicator light at the same position in the plurality of road images collected within a predetermined time, and also obtains the same position in the plurality of road images. The display state of the indicator lamp is determined based on a plurality of classification prediction results of the indicator lamp at the location.

In another embodiment, the plurality of road images are continuous frames of road images, and the determining unit 703 obtains the position in the first frame image of the continuous frame road image of the indicator light, and Based on the position of the indicator light in the image of the first frame and the movement speed and shooting frequency of the device for photographing the road image, each other angle other than the image of the first frame in the road image of the consecutive frames of the indicator light calculate a first position within the image of a frame, and also obtain a second position within the image of each other frame of the image of the successive frame of the indicator light, for the image of each frame in the image of the other frame, the If the difference between the second position and the first position is less than a predetermined value, it is determined that the indicator lights detected in the road images of the successive frames are the indicator lights of the same position.

In another embodiment, the display state of the indicator light includes always on or flashing, and the determining unit 703 is configured to: when the color classification prediction result of the indicator lamp at the same position in the plurality of road images is the same, the indicator lamp It is determined that the display state of the indicator is always on, and when the color classification prediction result of the indicator lamp at the same position in the plurality of images changes at intervals, it is determined that the display state of the indicator lamp is blinking.

8 is an indicator light detection device provided by at least one embodiment of the present invention, wherein the device includes a memory and a processor, the memory storing computer instructions executable on the processor, the processor executing the computer instructions; When executed, it realizes the indicator light detection method described in any embodiment of the present specification.

At least one embodiment of the present specification further provides a computer readable recording medium having a computer program stored thereon, and when the program is executed by a processor, the indicator light detection method described in any embodiment of the present specification is realized.

An embodiment of the present invention provides a computer program including computer readable code, and when the computer readable code is executed by a computer, the indicator light detection method described in any embodiment of the present invention is realized.

The description of each embodiment above tends to emphasize differences between the embodiments, and the same or similar parts may be referred to each other, and thus, for the sake of simplicity, the description is not repeated herein.

A person skilled in the art does not limit the implementation process in the above method of the specific embodiment, since the writing order of each step does not mean a strict execution order. The specific execution order of each step is determined by its function and possible internal logic.

In some embodiments, a function or module of an apparatus provided by an embodiment of the present invention implements the method described in the embodiment of the method, the specific realization may refer to the description of the embodiment of the method above. Therefore, it is not repeatedly described here for the sake of simplicity.

In an embodiment of the present invention, the computer-readable recording medium includes a plurality of forms, for example, in another example, the machine-readable recording medium is a RAM (Radom Access Memory, Random Access Memory), a volatile memory, and a non-volatile memory. It may be a volatile memory, a flash memory, a storage drive (eg hard disk drive), a solid state drive, any type of storage disk (eg CD, dvd, etc.), a similar recording medium, or a combination thereof. In a special case, the electrical computer readable medium may be paper or any suitable medium capable of printing a program. The medium may be used to acquire the program in an electrical manner (eg, optical scanning), and may be compiled, interpreted and processed according to a suitable method, and may be stored in a computer medium.

The above are only some embodiments of the present invention, and do not limit the present invention. Any modification, equivalent substitution, improvement, etc. made within the scope of the spirit and principle of the present invention should be included in the scope protected by the present invention.

Claims (17)

A method for detecting an indicator light, comprising:
recognizing the collected road images to obtain candidate bounding boxes of indicator lights in the road images;
predicting a plurality of classifications of indicator lights based on an image region corresponding to the candidate bounding box in the road image to obtain a plurality of classification prediction results of the indicator lights; , comprising at least two of a functional classification, a color classification, and an orientation classification; and
Comprising determining the display state of the indicator light based on a plurality of classification prediction results of the indicator light
Indicator light detection method, characterized in that. According to claim 1,
Predicting a plurality of classifications of an indicator light based on an image region corresponding to the candidate bounding box in the road image to obtain a plurality of classification prediction results of the indicator light,
performing feature extraction on an image region corresponding to the candidate bounding box using a neural network model to obtain an image feature corresponding to the candidate bounding box; and
Predicting a plurality of classifications of the indicator light by using an image feature corresponding to the candidate bounding box and a plurality of subnetwork branches included in the neural network model to obtain a plurality of classification prediction results of the indicator light,
The quantity of the plurality of subnetwork branches is equal to the quantity of the plurality of classifications, and each subnetwork branch is used to recognize a subtype of one of the plurality of classifications.
Indicator light detection method, characterized in that. 3. The method of claim 2,
Predicting a plurality of classifications of the indicator light using an image feature corresponding to the candidate bounding box and a plurality of subnetwork branches included in the neural network model to obtain a plurality of classification prediction results of the indicator light,
Predicting a first classification among a plurality of classifications of the indicator light using an image feature corresponding to the candidate bounding box and a first subnetwork branch among the plurality of subnetwork branches of at least two subtypes corresponding to the first classification to get predicted probabilities; and
and marking a subtype having the highest prediction probability among the at least two subtypes as a subtype of the first classification of the indicator light
Indicator light detection method, characterized in that. 4. The method according to any one of claims 1 to 3,
Determining the display state of the indicator light based on a plurality of classification prediction results of the indicator light,
When the plurality of classifications include the use classification and the shape classification, and the prediction result of the use classification is that the indicator light indicates a vehicle, and the prediction result of the shape classification is that the indicator light is a circular light, the arrangement combining the prediction results respectively corresponding to the classification, the function classification, and the color classification to obtain a first display state of the indicator light; or,
When the plurality of classifications include the use classification and the shape classification, and the predictive result of the use classification is that the indicator indicates a vehicle, and the predictive result of the shape classification is that the indicator light is an arrow, the color combining a classification and a prediction result respectively corresponding to the directed classification to obtain a second display state of the indicator light; or,
When the plurality of classifications include the use classification and the prediction result of the use classification is that the indicator light indicates a pedestrian, comprising obtaining a third display state of the indicator light
Indicator light detection method, characterized in that. 5. The method according to any one of claims 1 to 4,
Determining the display state of the indicator light based on a plurality of classification prediction results of the indicator light,
obtaining the indicator lights and a plurality of classification prediction results of the indicator lights at the same position in a plurality of road images collected within a predetermined time; and
determining the display state of the indicator light based on prediction results of a plurality of classifications of the indicator lights at the same position in the plurality of road images
Indicator light detection method, characterized in that. 6. The method of claim 5,
The plurality of road images are road images of consecutive frames,
Obtaining the indicator light at the same location in a plurality of road images collected within a predetermined time comprises:
obtaining the position of the indicator light within an image of a first frame of the road image of the successive frame;
Based on the position of the indicator light in the image of the first frame and the movement speed and shooting frequency of the device for photographing the road image, each other angle other than the image of the first frame in the road image of the consecutive frames of the indicator light calculating a first position within the image of the frame;
obtaining a second position within the image of each other frame of the image of the successive frame of the indicator light; and
With respect to the image of each frame among the images of other frames, if the difference between the second position and the first position is less than a predetermined value, it is determined that the indicator lights detected in the road images of the successive frames are the indicator lights of the same position including doing
Indicator light detection method, characterized in that. 7. The method of claim 5 or 6,
The display state of the indicator light includes always on or flashing,
Determining the display state of the indicator light based on a plurality of classification prediction results of the indicator lamp at the same position in the plurality of road images includes:
determining that the display state of the indicator lamp is always on when the color classification prediction results of the indicator lights at the same location in the plurality of road images are the same; and
Comprising determining that the display state of the indicator is blinking when the color classification prediction result of the indicator at the same position in the plurality of images changes at intervals
Indicator light detection method, characterized in that. An indicator light detection device comprising:
a recognition unit for recognizing the collected road image to obtain a candidate bounding box of an indicator light in the road image;
a prediction unit for predicting a plurality of classifications of indicator lights based on an image region corresponding to the candidate bounding box in the road image to obtain a plurality of classification prediction results of the indicator lights; and
a determining unit configured to determine a display state of the indicator light based on a plurality of classification prediction results of the indicator lamp;
The plurality of classifications include at least two of a use classification, a shape classification, an arrangement classification, a function classification, a color classification, and an orientation classification.
Indicator light detection device, characterized in that. 9. The method of claim 8,
The prediction unit is
performing feature extraction on the image region corresponding to the candidate bounding box using a neural network model to obtain image features corresponding to the candidate bounding box;
A plurality of classification prediction results of the indicator are obtained by predicting each of the plurality of classifications of the indicator using image features corresponding to the candidate bounding box and a plurality of subnetwork branches included in the neural network model,
The quantity of the plurality of subnetwork branches is equal to the quantity of the plurality of classifications, and each subnetwork branch is used to recognize a subtype of one of the plurality of classifications.
Indicator light detection device, characterized in that. 10. The method of claim 9,
The prediction unit is
Predicting a first classification among a plurality of classifications of the indicator light using an image feature corresponding to the candidate bounding box and a first subnetwork branch among the plurality of subnetwork branches of at least two subtypes corresponding to the first classification get the predicted probability,
Marking a subtype having the highest prediction probability among the at least two subtypes as a subtype of the first classification of the indicator light
Indicator light detection device, characterized in that. 11. The method according to any one of claims 8 to 10,
The confirmation unit is
When the plurality of classifications include the use classification and the shape classification, and the prediction result of the use classification is that the indicator light indicates a vehicle, and the prediction result of the shape classification is that the indicator light is a circular light, the arrangement a first display state of the indicator light is obtained by combining the prediction results respectively corresponding to the classification, the function classification and the color classification;
When the plurality of classifications include the use classification and the shape classification, and the prediction result of the use classification is that the indicator indicates a vehicle, and the predictive result of the shape classification is that the indicator light is an arrow, the color combining a classification and a prediction result respectively corresponding to the directed classification to obtain a second display state of the indicator light; or
When the plurality of classifications include the use classification and the prediction result of the use classification is that the indicator light indicates a pedestrian, obtaining a third display state of the indicator light
Indicator light detection device, characterized in that. 12. The method according to any one of claims 8 to 11,
The confirmation unit is
obtaining the indicator lights and a plurality of classification prediction results of the indicator lights at the same position in a plurality of road images collected within a predetermined time;
determining the display state of the indicator light based on prediction results of a plurality of classifications of the indicator lights at the same location in the plurality of road images
Indicator light detection device, characterized in that. 13. The method of claim 12,
The plurality of road images are road images of consecutive frames,
The confirmation unit is
obtain the position in the image of a first frame of the road image of the successive frame of the indicator light;
Based on the position of the indicator light in the image of the first frame and the movement speed and shooting frequency of the device for photographing the road image, each other angle other than the image of the first frame in the road image of the consecutive frames of the indicator light compute a first position within the image of the frame;
obtain a second position in the image of each other frame of the image of the successive frame of the indicator light;
With respect to the image of each frame among the images of the other frames, if the difference between the second position and the first position is less than a predetermined value, it is determined that the indicator lights detected in the road images of the successive frames are the indicator lights of the same position. to confirm
Indicator light detection device, characterized in that. 14. The method of claim 12 or 13,
The display state of the indicator light includes always on or flashing,
The confirmation unit is
If the color classification prediction results of the indicator lights at the same location in the plurality of road images are the same, it is determined that the display state of the indicator lights is always on,
When the color classification prediction result of the indicator lamp at the same position in the plurality of images changes at intervals, determining the display state of the indicator lamp as blinking
Indicator light detection device, characterized in that. An indicator light detection device comprising:
a memory and a processor;
the memory stores computer instructions executable on the processor;
The processor realizes the method according to any one of claims 1 to 7 when the computer instructions are executed.
Indicator light detection device, characterized in that. A computer-readable recording medium having a computer program stored therein, comprising:
The method according to any one of claims 1 to 7 is realized when the computer program is executed by a processor.
A computer-readable recording medium, characterized in that. A computer program comprising computer readable code and stored in a recording medium,
The method according to any one of claims 1 to 7 is realized when the computer readable code is executed by a computer.
A computer program stored in a recording medium, characterized in that.

Images