CN111723625A - Traffic light image recognition processing method and device, auxiliary traffic system and storage medium - Google Patents
Traffic light image recognition processing method and device, auxiliary traffic system and storage medium Download PDFInfo
- Publication number
- CN111723625A CN111723625A CN201910222479.6A CN201910222479A CN111723625A CN 111723625 A CN111723625 A CN 111723625A CN 201910222479 A CN201910222479 A CN 201910222479A CN 111723625 A CN111723625 A CN 111723625A
- Authority
- CN
- China
- Prior art keywords
- image
- traffic light
- template
- information
- template library
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
- G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
- G08G1/09623—Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a traffic light image identification processing method, which comprises the following steps: s100: acquiring an image containing depth information and color information and preprocessing the image; s200: positioning a traffic light background frame in the image; s300: identifying the type and color of the traffic light in the background frame; s400: and tracking the target of the traffic light, and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition. The invention also provides a traffic light image recognition processing device, an auxiliary traffic system and a computer readable storage medium. After the technical scheme is adopted, the traffic lights can be accurately identified, the risk of misinformation is obviously reduced, and effective assistance is provided for the passing of drivers.
Description
Technical Field
The invention relates to the technical field of auxiliary traffic, in particular to a traffic light image identification processing method, a traffic light image identification processing device, an auxiliary traffic system and a storage medium.
Background
In the prior art, the intersection auxiliary passing system generally comprises a camera arranged in the center of the upper part of a front windshield of an automobile, the camera is connected with a controller in the automobile through a wire, the controller is also connected with a voice alarm through a wire, the controller acquires images of a traffic light in front through the camera, identifies the color of the traffic light, and then controls the voice alarm to send out a corresponding voice command according to the identified color.
However, in the prior art, the traffic light target is generally processed only by identification, that is, the camera acquires an image containing road conditions of the intersection, the vehicle-mounted embedded processor processes the acquired single-frame image, and then performs early warning judgment, and sends early warning information to the user according to a judgment result, so that the early warning information is used as a means for assisting in driving the automobile. However, the method has obvious risk of false alarm, and no better solution is provided for complex conditions such as color switching of traffic lights, rapid distance of automobiles from intersections and the like. Most systems are combined with a signal generating device arranged on a traffic light at an intersection, when a vehicle passes through the intersection, the device sends traffic light state information to the vehicle, and the vehicle acquires the information and then performs judgment processing by combining self state data such as vehicle speed and the like to realize an early warning function, so that the complexity of implementation of a scheme is increased undoubtedly.
Therefore, it is necessary to develop a traffic light image recognition processing method, a traffic light image recognition processing device, an auxiliary traffic system, and a storage medium, which can accurately recognize traffic lights and reduce the risk of false alarms.
Disclosure of Invention
In order to overcome the technical defects, the invention aims to provide a traffic light image recognition processing method, a traffic light image recognition processing device, an auxiliary traffic system and a storage medium, wherein the traffic light image recognition processing method, the traffic light image recognition processing device, the auxiliary traffic system and the storage medium can accurately recognize traffic lights and reduce the risk of false alarm.
The invention discloses an image identification processing method for a traffic light, which comprises the following steps:
s100: acquiring an image containing depth information and color information and preprocessing the image;
s200: positioning a traffic light background frame in the image;
s300: identifying the type and color of the traffic light in the background frame;
s400: and tracking the target of the traffic light, and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
Preferably, the step S400 includes:
s401: analyzing the first frame image after the steps S100-S300, extracting a characteristic vector, and storing the characteristic vector as a template into a template base;
s402: extracting a characteristic vector from the image in the subsequent frame after the step S100-S300, and then matching the characteristic vector with a template library;
s403: updating the template base according to the matching result;
s404: and when the number of the frames which are successfully matched continuously exceeds the set number of the frames, outputting the state information and the distance information of the traffic lights.
Preferably, the step S401 specifically includes:
analyzing the image of the first frame to obtain the mean value and the variance of the gray values in the preset range near the characteristic points and the distribution information of the surrounding characteristic points;
establishing attribute vectors of the feature points according to the analyzed information, and storing the attribute vectors as templates in a template library;
the step S402 specifically includes:
analyzing the images in the subsequent frames, establishing attribute vectors of the feature points, matching the attribute vectors with templates in a template library, and comparing the similarity;
the step S403 specifically includes:
when the attribute vector of the image feature point in the subsequent frame is completely the same as any template in the template library, covering the template in the template library; when the attribute vector of the image feature point in the subsequent frame is not completely the same as the template of the template library, storing the attribute vector as a new template into the template library;
defining the reliability weight of the template according to the use frequency of the template, and updating the template in the template library according to the reliability weight when a new template is stored in the template library;
the step S404 specifically includes:
comparing the attribute vector of the image feature point in the subsequent frame with the reliability weight threshold of the template library, and judging that the matching is successful when the attribute vector of the image feature point in the subsequent frame is greater than the reliability weight threshold of the template library;
when the number of frames which are successfully matched continuously exceeds the set number of frames, outputting the state information and the distance information of the traffic light of the current frame;
when the traffic light is a round traffic light, the state information is the color of the traffic light; when the traffic light is an arrow traffic light, the state information is the direction and the color of the arrow;
the distance information is depth information of a traffic light region on the image.
Preferably, the attribute vector is:
V＝(m1，m2，μ，σ，p0，p1，…，p8，)
m1the distance of the position where other characteristic points are distributed most in a preset range around the characteristic point is obtained; m is2The distance of other characteristic points distributed for a plurality of times in a preset range around the characteristic point is obtained; mu is the gray variance of 9 pixels around the feature point, sigma is the gray standard deviation of 9 pixels around the feature point, p0，p1，...，p8The 9 pixel points around the feature point are arranged from large to small according to the gray level.
Preferably, the reliability weight of the template in the template library at time t is defined as:
u is unit step function, β is coefficient vector which is the product of attribute vector of feature point and use frequency of template, βiThe coefficient corresponding to the ith template is obtained by the coefficient vector through the cosine theorem; n is the total number of the templates;
the reliability weight threshold of the template library is the product of the average value of the reliability weights of the templates in the template library and a preset coefficient;
updating the template in the template library according to the reliability weight, specifically comprising: when the number of the templates in the template library does not reach the total number N, deleting the templates with the reliability weights lower than the reliability weight threshold of the template library when the new templates are stored in the template library; when the number of templates in the template library reaches the total number N and templates with similarity lower than the reliability weight threshold of the template library exist, deleting the templates with the reliability weights lower than the reliability weight threshold of the template library when a new template is stored in the template library; and when the number of the templates in the template library reaches the total number N and the similarity of the templates is higher than the reliability weight threshold of the template library, randomly extracting one template with the reliability weight higher than the reliability weight threshold of the template library for updating.
Preferably, the step S400 is followed by the following steps:
s500: acquiring the driving information of the vehicle, and sending out reminding information when the driving information of the vehicle and the color and distance of the output traffic light meet preset conditions;
the travel information includes: one or more of vehicle speed information, steering lamp information, wiper information and gear information;
the reminding information comprises: voice prompt, light prompt and vibration prompt.
The invention also discloses a traffic light image recognition processing device, which comprises:
the acquisition module is used for acquiring an image containing depth information and color information and preprocessing the image;
the positioning module is used for positioning the traffic light background frame in the image;
the identification module is used for identifying the type and the color of the traffic light in the background frame;
and the tracking module is used for tracking the traffic light and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
The invention also discloses an auxiliary passing system, which comprises:
the image acquisition module is used for acquiring an image containing depth information and color information;
the whole vehicle communication module is communicated with a vehicle body CAN network and used for acquiring the running information of the vehicle;
the data processing module is in communication connection with the image acquisition module and the vehicle communication module and comprises a processor, a memory and an I/O port, wherein the memory is used for storing a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions and execute the image identification processing method;
and the prompt early warning module is communicated with the whole vehicle communication module through a vehicle body CAN network and is used for sending out prompt information.
Preferably, the image acquisition module comprises a 3D camera having a CMOS image sensor combined with a TOF;
the prompt early warning module comprises one or more of a loudspeaker, an eccentric motor and an alarm lamp.
The invention also discloses a computer-readable storage medium, in which a computer program is stored, the computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the image recognition processing method described above.
After the technical scheme is adopted, compared with the prior art, the method has the following beneficial effects:
1. the traffic light can be accurately identified and the risk of false alarm is obviously reduced;
2. the search space in the template matching process cannot be obviously increased along with the increase of the size of the image and the size of the template, the complexity of algorithm calculation is reduced, and the target tracking efficiency is improved.
3. Target reconstruction is added before a new frame of image is matched, the region where the target is likely to appear is predicted, the searching range is narrowed, the iteration times of tracking operation are reduced to a certain extent, and the real-time performance of the system is improved.
Drawings
FIG. 1 is a flow chart of an image recognition processing method according to an embodiment of the present invention;
FIG. 2 is a flowchart of an image recognition processing method according to an embodiment of the present invention
Fig. 3 is a schematic structural diagram of an auxiliary traffic system according to an embodiment of the present invention.
Reference numerals:
100-an image acquisition module, 120-a data processing module, 130-a vehicle communication module and 140-a prompt early warning module.
Detailed Description
The advantages of the invention are further illustrated in the following description of specific embodiments in conjunction with the accompanying drawings.
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
In the description of the present invention, it is to be understood that the terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used merely for convenience of description and for simplicity of description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed in a particular orientation, and be operated, and thus, are not to be construed as limiting the present invention.
In the description of the present invention, unless otherwise specified and limited, it is to be noted that the terms "mounted," "connected," and "connected" are to be interpreted broadly, and may be, for example, a mechanical connection or an electrical connection, a communication between two elements, a direct connection, or an indirect connection via an intermediate medium, and specific meanings of the terms may be understood by those skilled in the art according to specific situations.
In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "module" and "component" may be used in a mixture.
Referring to fig. 1, the present invention provides a traffic light image recognition processing method, which includes the following steps:
s100: and acquiring an image containing depth information and color information and preprocessing the image.
Specifically, the step S100 includes:
s101: and acquiring an image containing depth information and color information through an image acquisition module arranged on the vehicle. The image acquisition module may be a 3D camera disposed on a vehicle. Preferably, a CMOS (complementary Metal Oxide semiconductor) image sensor combined with tof (time of flight) is arranged in the 3D camera, the image sensor can simultaneously obtain a depth image and a common RGB color image, each pixel unit in the sensor comprises 1Z pixel (for generating the depth image) and 8 RGB pixels (for generating the RGB color image), and when the same set of optical lenses are used, the CMOS sensor almost simultaneously generates an RGB color image and a depth image.
S102: and performing histogram equalization operation on the color image. The color tone of the image is enhanced through the histogram equalization operation, thereby facilitating the identification of the color. Preferably, before the step S102, the method further comprises an image extraction step, wherein the step S102 is carried out on the image obtained in the step S101 by extracting the area of the image from about 1/3-1/2. By such processing, the region of interest of the image processing can be reduced, the data amount of the processing can be reduced, and the influence of a part of other light sources (such as a part of automobile tail lights) can be reduced.
S103: the image obtained in step S102 is denoised. Preferably, the denoising is performed using gaussian filtering.
S104: the color of the large block area in the image obtained in step S103 is filtered out by using a morphological processing method, and the color of the bright small block part is retained. At this time, the obtained image is subjected to preprocessing.
After step S100 is completed, the steps are performed:
s200: and positioning the background frame of the traffic light in the image.
Specifically, step S200 includes:
s201: and converting the image obtained in the step S100 into an HSV color space, and segmenting the image by utilizing the characteristics of the red, green and yellow colors to obtain a target segmentation image with the characteristics of the red, green and yellow colors.
S202: and analyzing the depth information of the image in the step S201, extracting a target area in a set depth range, and performing morphological processing on the target area. Because the acquired image contains depth information and color information, namely each pixel unit in the image has distance information, proper screening is carried out according to a preset depth threshold range, and some interference areas can be excluded. One possible setting of the preset depth threshold range is as follows: according to the GB 14886-2006 road traffic signal lamp setting and installation specification, a target region which is too close or too far in an image is an interference region, and a region within a proper range (such as a range of 50-200 m) is a region in which a target image needs to be extracted. Meanwhile, the installation angle of an image acquisition module (such as a 3D camera) can be combined, and the area with an excessively large or excessively small angle in the image can be discharged. And performing morphological processing on the extracted target image area to remove noise interference in the image. For example, morphological opening operation processing is performed on the target image to remove fine protrusions (such as connection between a traffic light background frame and a traffic light pole) in the image.
S203: the image obtained in S202 is subjected to background detection, and a rectangular black frame having a squareness or an aspect ratio within a preset range is detected. The value of the rectangular degree R is between 0 and 1. When the object is rectangular, R takes a maximum value of 1. A rectangle degree threshold (e.g. 0.9) may be set, that is, when the rectangle degree of the background area is greater than the set rectangle degree threshold, the background area is considered to be a rectangular background frame, and areas that do not satisfy the rectangle degree threshold may be excluded. Thus, the positioning of the traffic light background frame is realized.
After step S200 is completed, the steps are performed:
s300: and identifying the type and the color of the traffic light in the background frame.
Specifically, in step S300, the type of traffic light is first determined according to the shape of the traffic light within the traffic light background frame. And segmenting candidate areas in the traffic light background frame, and detecting the aspect ratio of the candidate areas. The aspect ratio of a traffic light such as a circle should ideally be 1: 1, if the length and width ratios of the candidate regions in the segmented image are very different, it can be determined that this region is not a circular traffic light. Setting the length-width ratio of the area to be 0.7-1.4, if the area is considered to be a circular traffic light, and performing step S310; if the area is not located in the area, the area is determined not to be a circular traffic light, and the area is recognized as an arrow traffic light, and the process proceeds to step S320.
S310: a circular traffic light identification comprising the steps of:
s311: performing binarization processing on the image obtained based on the step 300 by adopting an Otsu (maximum inter-class variance method);
s312: converting the color image containing the traffic light background frame area from an RGB color space to an Lab color space, then segmenting the color image by utilizing the characteristics of L, a and b components in the Lab color space and the information of the image obtained in the step S311 to obtain an initial segmentation image, and judging the color type through the selection range of the traffic light color based on a and b. By way of example, all possible light colors of a round traffic light are red, green and yellow. Wherein, the threshold range of green is: -50< a < -8 and 15< b < 80; the threshold range for red is 15< a <110 and 15< b < 60; the threshold range for yellow is 1< a <16 and 25< b < 60. Thus, in this step, the color of the traffic light can be judged.
S313: the depth information of the image obtained in S312 is analyzed, a target region within the set depth range is extracted, and morphological processing is performed on the target region to exclude regions that do not meet the requirements. This step is similar to step S202 and will not be described here.
S314: and calculating the region of interest to obtain the minimum circumscribed rectangle of the region of interest. The region of interest refers to a region to be processed which is delineated from the processed image in a manner of a box, a circle, an ellipse, an irregular polygon and the like. In image processing, generally, an ROI of an image is obtained by an operator and a function, a calculation range is narrowed, and then the image is processed in the next step. This area is the focus of your image analysis. The area is delineated for further processing. The ROI is used for delineating the target which the user wants to read, so that the processing time can be reduced, and the precision can be increased. The area of interest refers to an area with an aspect ratio meeting the requirement of a round traffic light in a traffic light background frame.
S315: the method comprises the steps of solving the Hu characteristics of a template of the circular traffic light and an image of the interested region by using seven functions of the Hu invariant moment, matching the characteristic values of the interested regions by using the solved template characteristic values, and if the obtained values are smaller, indicating that the obtained values are more similar. And when the value of the similarity is within a preset range, judging the lamp to be a circular lamp. The template refers to templates of various types of traffic lights in the template library. This step is for accurately identifying the circular traffic light.
S320: the method for recognizing the arrow traffic light comprises the following steps:
s321: the image obtained in step S310 is converted into an HSV color space. Segmenting the image by utilizing the characteristics of red, green and yellow colors to obtain a target segmentation image with the characteristics of the red, green and yellow colors;
s322: carrying out gray level processing on the background frame partial image of the traffic light;
s323: carrying out binarization processing on the obtained gray level image, carrying out regional automatic threshold binarization processing on the obtained gray level image by combining a color segmentation image, and separating an arrow traffic light;
s324: performing closed operation on the obtained binary image;
s325: extracting a minimum arrow target area and carrying out normalization processing;
s326: the arrow type identification is performed using the characteristics of the arrow traffic light itself. The characteristic of the traffic light itself is the symmetry of the arrow. For example, when the arrow traffic light is symmetric about the horizontal direction symmetry axis and is not symmetric about the vertical direction symmetry axis, determining that the arrow of the arrow traffic light points to the left or the right; if the number of target points (the target points are pixel points forming the arrow traffic light) positioned on the left side of the vertical direction symmetry axis is larger than that of the target points positioned on the right side of the vertical direction symmetry axis, determining that the type of the arrow traffic light is a left-going straight line; and if the number of the target points positioned on the left side of the vertical direction symmetry axis is less than that of the target points positioned on the right side of the vertical direction symmetry axis, determining that the arrow type of the arrow traffic light is moving straight to the right. In the step, the arrow direction information of the arrow traffic light can be determined, and the light color of the arrow traffic light can be obtained by combining the result of color segmentation.
S400: and tracking the target of the traffic light, and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
Specifically, S400 includes the steps of:
s401: and (5) analyzing the first frame image after the steps S100-S300, extracting a characteristic vector, and storing the characteristic vector as a template into a template base.
The images of consecutive frames can be obtained by an image obtaining module (e.g. a 3D camera), and steps S100 to S300 are performed. Analyzing the image of the first frame after the step S100-S300 to obtain the mean value and the variance of the gray values in the preset range near the characteristic points and the distribution information of the surrounding characteristic points; and establishing an attribute vector of the feature point according to the analyzed information, and storing the attribute vector as a template into a template library.
In image processing, a feature point refers to a point where the image gradation value changes drastically or a point where the curvature is large on an image edge (i.e., an intersection of two edges). The image feature points play an important role in the image matching algorithm based on the feature points. The image feature points can reflect the essential features of the image and can identify the target object in the image. Matching of images can be completed through matching of feature points.
In particular, attribute vectors of feature points
V＝(m1，m2，μ，σ，p0，p1，…，p8，)
m1The distance of the position where other characteristic points are distributed most in a preset range around the characteristic point is obtained; m is2The distance of other characteristic points distributed for a plurality of times in a preset range around the characteristic point is obtained; mu is the gray variance of 9 pixels around the feature point, sigma is the gray standard deviation of 9 pixels around the feature point, p0，p1，...，p8The 9 pixel points around the feature point are arranged from large to small according to the gray level.
S402: and (5) extracting a characteristic vector from the image in the subsequent frame after the steps S100-S300, and then matching the characteristic vector with a template library.
And (5) analyzing the images in the subsequent frames after the steps S100-S300, establishing an attribute vector V of the feature point, matching the attribute vector V with the template in the template library, and comparing the similarity. Further, in this step, the relationship between the coordinates of the target center point and the boundary of the tracking area is analyzed, it is determined whether the target object tends to be stationary and cannot be tracked or the target has been left from the tracking area and stopped tracking, if the target center point is located in the tracking area, we call template matching for searching, and for each frame, it is estimated that one current observation sequence is tracked from the beginning.
S403: and updating the template base according to the matching result.
Specifically, when the attribute vector of the image feature point in the subsequent frame is completely the same as any template in the template library, covering the template in the template library; and when the attribute vector of the image feature point in the subsequent frame is not completely the same as the template of the template library, storing the attribute vector as a new template into the template library.
For the templates in the template library, the reliability weights are defined according to the use frequency of the templates. The usage frequency of the template refers to the number of times the corresponding template in the template library is covered by the new template in a unit time. The reliability weight of the template at the time t is defined as
Where U is a unit step function, β is a coefficient vector, which is the product of the attribute vector of the feature point and the use frequency of the template, βiThe coefficient corresponding to the ith template is obtained by the coefficient vector through the cosine theorem; n is the total number of templates. The template coefficient is a mathematical distance, which can be used to measure the similarity between two coefficient vectors, and the relationship between the template coefficient and the coefficient vector can be obtained by the cosine theorem. Such as:
cos(A)＝(b1c1+b2c2+…bncn)/sqrt[(b1 2+b2 2+…+bn 2)(c1 2+c2 2+…+cn 2)]
where the numerator represents the inner product of two vectors and the denominator represents the modulo multiplication of the two vectors.
In order to ensure the diversity of the templates, when a new template is stored in the template library, one template with the reliability weight higher than the set threshold can be randomly selected for updating, and the template with the reliability weight lower than the set threshold can be deleted. The set threshold referred to herein is a reliability weight threshold of the template library. The reliability weight threshold of the template library may be obtained by multiplying an average of the reliability weights of the templates in the template library by a preset coefficient, where the preset coefficient is less than 1, and is preferably 3/5. According to the formula, the reliability weight value changes after the template base is updated, and correspondingly, the threshold value of the reliability weight value also changes. Preferably, when the number of the templates in the template library does not reach the total number N, deleting the templates with the reliability weights lower than a set threshold value when the new templates are stored in the template library; when the number of templates in the template library reaches the total number N and templates with similarity lower than a set threshold exist, deleting the templates with reliability weights lower than the set threshold when a new template is stored in the template library; and when the number of the templates in the template library reaches the total number N and the similarity of the templates is higher than a set threshold, randomly extracting one template with the reliability weight higher than the set threshold for updating.
S404: and when the number of the frames which are successfully matched continuously exceeds the set number of the frames, outputting the state information and the distance information of the traffic lights.
And comparing the attribute vector of the image feature point in the subsequent frame with the reliability weight threshold of the template library, and judging that the matching is successful when the attribute vector of the image feature point in the subsequent frame is greater than the reliability weight threshold of the template library.
When the number of frames successfully matched continuously exceeds the set number of frames, preferably, the set number of frames is 10 frames, and the state information and the distance information of the traffic light of the current frame are output. When the traffic light is a round traffic light, the state information is the color of the traffic light; when the traffic light is an arrow traffic light, the state information is the direction and the color of the arrow; the distance information is depth information of a traffic light region on the image, namely distance information from the traffic light to an image acquisition module (such as a 3D camera). The state information of the traffic light is obtained in step S300, and the distance information may be directly obtained through the depth image acquired by the image acquisition module. The state information and the distance information of the traffic lights can be output to the outside, and can also be output to a subsequent program for processing. And when the number of frames which are successfully matched does not meet the preset number of frames, not outputting the state information and the distance information of the traffic light.
Referring to fig. 2, step S400 preferably further includes the following steps:
s500: acquiring the driving information of the vehicle, and sending out reminding information when the driving information of the vehicle and the color and distance of the output traffic light meet preset conditions; the travel information includes: one or more of vehicle speed information, steering lamp information, wiper information and gear information; the reminding information comprises: voice prompt, light prompt and vibration prompt.
Specifically, vehicle running information required for intersection auxiliary judgment, namely real-time data of the whole vehicle, such as one or more of vehicle speed information, steering lamp information, wiper information and gear information, is acquired through a vehicle body CAN network. And then, the determination is made in conjunction with the state information and the distance information of the traffic lights output in step S400.
For example, if the automobile runs straight and is in a low-speed running state, the traffic light is in a red light (red arrow or red round light) state, and the depth information of the traffic light (i.e., the distance from the automobile to the traffic light) is smaller than a preset low-speed alarm threshold value, an early-warning message signal is sent to the entire automobile CAN network, a mechanical motor installed in a steering wheel emits low-frequency vibration, an alarm light in the automobile is always on, and an internal speaker plays a warning voice to remind a driver.
If the automobile runs straight and is in a high-speed running state, the traffic light is in a red light (red arrow or red round light) state, and the depth information (namely the distance from the automobile to the traffic light) of the traffic light is smaller than a preset high-speed alarm threshold value, an early-warning message signal is sent to a finished automobile CAN network, the finished automobile sends a control instruction, a mechanical motor arranged in a steering wheel sends high-frequency vibration, and an alarm light flickers; the built-in loudspeaker plays alarm voice to remind the driver.
If the automobile turns on the left steering lamp, the traffic light is in a red light (red left steering arrow or red round light) state of 10 frames or more, and the depth information (namely the distance from the automobile to the traffic light) of the traffic light is smaller than a preset low-speed alarm threshold value, an early-warning message signal is sent to a finished automobile CAN network, a mechanical motor arranged in a steering wheel sends out high-frequency vibration, and an alarm voice prompt is generated in the automobile.
If the automobile moves straight and the traffic light is in a yellow light (red arrow or round light) flashing state, an early warning message signal is sent to the entire automobile CAN network slightly and the warning light flashes.
Preferably, when it is acquired that the wiper is opened for more than the preset time, the weather is considered as rainy days, and the value of the alarm threshold is appropriately adjusted to adapt to the weather.
The above-mentioned judgment rules and reminding modes are only preferred, and can be flexibly set according to the needs.
According to the technical scheme, the traffic light is tracked, and the traffic state information and the distance information are output for subsequent processing after the traffic light image is successfully matched with the template in the template library for the preset frame number, so that the accuracy of traffic light identification is improved, and the risk of misinformation is remarkably reduced. Meanwhile, the traffic light in the traffic light background frame is tracked, so that the search space in the template matching process is not obviously increased along with the increase of the size of the image and the size of the template, the complexity of algorithm calculation is reduced, and the target tracking efficiency is improved. Meanwhile, target reconstruction is added before a new frame of image is matched, the region where the target is likely to appear is predicted, the searching range is narrowed, the iteration times of tracking operation are reduced to a certain extent, and the real-time performance of the system is improved.
The invention also provides a traffic light image recognition processing device, which comprises:
the acquisition module is used for acquiring an image containing depth information and color information and preprocessing the image;
the positioning module is used for positioning the traffic light background frame in the image;
the identification module is used for identifying the type and the color of the traffic light in the background frame;
and the tracking module is used for tracking the traffic light and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
Preferably, the image recognition processing apparatus further includes:
and the judgment reminding module is used for acquiring the driving information of the vehicle and sending out reminding information when the driving information of the vehicle and the color and distance of the output traffic light meet preset conditions.
Referring to fig. 3, the present invention also provides an auxiliary passage system, comprising:
For obtaining an image containing depth information and color information. The image acquisition module 110 preferably comprises a 3D camera. The 3D camera is internally provided with a CMOS (complementary Metal oxide semiconductor) image sensor combined with TOF (time of flight), a depth image and a common RGB (Red, Green, blue) color image can be obtained simultaneously, each pixel unit in the sensor comprises 1Z pixel (used for generating the depth image) and 8 RGB pixels (used for generating the RGB color image), and when the same set of optical lenses are used, the CMOS sensor almost simultaneously generates an RGB color image and a depth image. The image capturing module 110 is installed on a vehicle and is used for capturing an image of a real-time road condition of a road in front of the vehicle.
The vehicle body CAN network is communicated with the vehicle body CAN network and is used for acquiring the running information of the vehicle;
a data processing module 120
The data processing module 120 includes a processor, a memory and an I/O port, the memory is used for storing a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the image recognition processing method. Specifically, the processor is an embedded microprocessor, and the memory comprises a read-only memory which completes power-on self-test of the system, initialization of each functional module, operation and guidance of input/output driving of the system, and a random access memory which can be read and written at any time and is used for storing temporary data in an operating program. The computer program is stored in a read-only memory. The I/O port can convert information formats, set interrupt and DMA (direct Memory access) control logic, and realize information exchange between a DSP (digital Signal processing) and peripheral equipment in an interrupt mode.
Prompt early warning module 140
And the vehicle communication module 130 communicates with the vehicle body through a vehicle body CAN network and is used for sending out reminding information. Specifically, the prompt and early warning module 140 includes one or more of a speaker, an eccentric motor, and a warning lamp. The loudspeaker is used for playing voice so as to remind the driver; the eccentric motor is arranged in the steering wheel and used for generating vibration to remind a driver; the warning lamp is used for reminding the driver in a flashing or normally-on mode. The data processing module 120 sends out a pre-warning message signal to a vehicle CAN network (CAN bus) through the vehicle communication module 130; the prompt and early warning module 140 receives the pre-warning message signal through the vehicle body CAN network and sends out corresponding prompt according to the pre-warning message signal.
The present invention also provides a computer-readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the image recognition processing method described above. The computer readable storage medium may be a read-only memory, a magnetic or optical disk, or the like.
It should be noted that the embodiments of the present invention have been described in terms of preferred embodiments, and not by way of limitation, and that those skilled in the art can make modifications and variations of the embodiments described above without departing from the spirit of the invention.
Claims (10)
1. A traffic light image identification processing method comprises the following steps:
s100: acquiring an image containing depth information and color information and preprocessing the image;
s200: positioning a traffic light background frame in the image;
s300: identifying the type and color of the traffic light in the background frame;
it is characterized by also comprising the following steps,
s400: and tracking the target of the traffic light, and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
2. The image recognition processing method according to claim 1,
the step S400 includes:
s401: analyzing the first frame image after the steps S100-S300, extracting a characteristic vector, and storing the characteristic vector as a template into a template base;
s402: extracting a characteristic vector from the image in the subsequent frame after the step S100-S300, and then matching the characteristic vector with a template library;
s403: updating the template base according to the matching result;
s404: and when the number of the frames which are successfully matched continuously exceeds the set number of the frames, outputting the state information and the distance information of the traffic lights.
3. The image recognition processing method according to claim 2,
the step S401 specifically includes:
analyzing the image of the first frame to obtain the mean value and the variance of the gray values in the preset range near the characteristic points and the distribution information of the surrounding characteristic points;
establishing attribute vectors of the feature points according to the analyzed information, and storing the attribute vectors as templates in a template library;
the step S402 specifically includes:
analyzing the images in the subsequent frames, establishing attribute vectors of the feature points, matching the attribute vectors with templates in a template library, and comparing the similarity;
the step S403 specifically includes:
when the attribute vector of the image feature point in the subsequent frame is completely the same as any template in the template library, covering the template in the template library; when the attribute vector of the image feature point in the subsequent frame is not completely the same as the template of the template library, storing the attribute vector as a new template into the template library;
defining the reliability weight of the template according to the use frequency of the template, and updating the template in the template library according to the reliability weight when a new template is stored in the template library;
the step S404 specifically includes:
comparing the attribute vector of the image feature point in the subsequent frame with the reliability weight threshold of the template library, and judging that the matching is successful when the attribute vector of the image feature point in the subsequent frame is greater than the reliability weight threshold of the template library;
when the number of frames which are successfully matched continuously exceeds the set number of frames, outputting the state information and the distance information of the traffic light of the current frame;
when the traffic light is a round traffic light, the state information is the color of the traffic light; when the traffic light is an arrow traffic light, the state information is the direction and the color of the arrow;
the distance information is depth information of a traffic light region on the image.
4. The image recognition processing method according to claim 3,
the attribute vector is:
V＝(m1，m2，μ，σ，p0，p1，...，p8，)
m1the distance of the position where other characteristic points are distributed most in a preset range around the characteristic point is obtained; m is2The distance of other characteristic points distributed for a plurality of times in a preset range around the characteristic point is obtained; mu is the gray variance of 9 pixels around the feature point, sigma is the gray standard deviation of 9 pixels around the feature point, p0，p1，...，p8The 9 pixel points around the feature point are arranged from large to small according to the gray level.
5. The image recognition processing method according to claim 3,
the reliability weight of the template in the template library at the time t is defined as:
u is unit step function, β is coefficient vector which is the product of attribute vector of feature point and use frequency of template, βiThe coefficient corresponding to the ith template is obtained by the coefficient vector through the cosine theorem; n is the total number of the templates;
the reliability weight threshold of the template library is the product of the average value of the reliability weights of the templates in the template library and a preset coefficient;
updating the template in the template library according to the reliability weight, specifically comprising: when the number of the templates in the template library does not reach the total number N, deleting the templates with the reliability weights lower than the reliability weight threshold of the template library when the new templates are stored in the template library; when the number of templates in the template library reaches the total number N and templates with similarity lower than the reliability weight threshold of the template library exist, deleting the templates with the reliability weights lower than the reliability weight threshold of the template library when a new template is stored in the template library; and when the number of the templates in the template library reaches the total number N and the similarity of the templates is higher than the reliability weight threshold of the template library, randomly extracting one template with the reliability weight higher than the reliability weight threshold of the template library for updating.
6. The image recognition processing method according to claim 1,
the step S400 is followed by the following steps:
s500: acquiring the driving information of the vehicle, and sending out reminding information when the driving information of the vehicle and the color and distance of the output traffic light meet preset conditions;
the travel information includes: one or more of vehicle speed information, steering lamp information, wiper information and gear information;
the reminding information comprises: voice prompt, light prompt and vibration prompt.
7. A traffic light image recognition processing apparatus, characterized by comprising:
the acquisition module is used for acquiring an image containing depth information and color information and preprocessing the image;
the positioning module is used for positioning the traffic light background frame in the image;
the identification module is used for identifying the type and the color of the traffic light in the background frame;
and the tracking module is used for tracking the traffic light and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
8. An assisted entry system, comprising:
the image acquisition module is used for acquiring an image containing depth information and color information;
the whole vehicle communication module is communicated with a vehicle body CAN network and used for acquiring the running information of the vehicle;
a data processing module communicatively connected to the image acquisition module and the vehicle communication module, the data processing module comprising a processor, a memory, and an I/O port, the memory for storing a computer program, the computer program comprising program instructions, the processor configured to invoke the program instructions to perform the method of any one of claims 1-6;
and the prompt early warning module is communicated with the whole vehicle communication module through a vehicle body CAN network and is used for sending out prompt information.
9. An auxiliary traffic system as claimed in claim 8,
the image acquisition module comprises a 3D camera, the 3D camera having a CMOS image sensor combined with a TOF;
the prompt early warning module comprises one or more of a loudspeaker, an eccentric motor and an alarm lamp.
10. A computer-readable storage medium, characterized in that,
the computer storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910222479.6A CN111723625B (en) | 2019-03-22 | 2019-03-22 | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910222479.6A CN111723625B (en) | 2019-03-22 | 2019-03-22 | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111723625A true CN111723625A (en) | 2020-09-29 |
| CN111723625B CN111723625B (en) | 2023-08-08 |
Family
ID=72562802
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910222479.6A Active CN111723625B (en) | 2019-03-22 | 2019-03-22 | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111723625B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112150364A (en) * | 2020-10-10 | 2020-12-29 | 厦门理工学院 | Pairing and splicing method for split type candidate image areas of arrow-shaped traffic signal lamp |
| CN112528794A (en) * | 2020-12-03 | 2021-03-19 | 北京百度网讯科技有限公司 | Signal lamp fault identification method and device and road side equipment |
| CN113053152A (en) * | 2021-03-08 | 2021-06-29 | 江铃汽车股份有限公司 | Vehicle-mounted traffic sign real-time display method and system |
| CN113052190A (en) * | 2021-04-01 | 2021-06-29 | 威海职业学院(威海市技术学院) | Visual auxiliary system and method based on image processing |
| CN113989774A (en) * | 2021-10-27 | 2022-01-28 | 广州小鹏自动驾驶科技有限公司 | Traffic light detection method and device, vehicle and readable storage medium |
| CN116152784A (en) * | 2023-04-21 | 2023-05-23 | 深圳市夜行人科技有限公司 | Signal lamp early warning method and system based on image processing |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20070074913A (en) * | 2006-01-11 | 2007-07-18 | 주식회사 현대오토넷 | System and method for reporting change of signal lamp using image recognition |
| KR101096806B1 (en) * | 2010-10-13 | 2011-12-22 | 한국전자통신연구원 | System for informing traffic signal |
| CN102800109A (en) * | 2012-07-16 | 2012-11-28 | 刘怡光 | Method for tracking locomotive signal lamp based on particle filter |
| CN102819728A (en) * | 2012-07-17 | 2012-12-12 | 中国航天科工集团第三研究院第八三五七研究所 | Traffic sign detection method based on classification template matching |
| DE102013101175A1 (en) * | 2012-02-13 | 2013-08-14 | Toyota Motor Engineering & Manufacturing North America Inc. | System and method for detecting traffic signals |
| CN105185140A (en) * | 2015-09-30 | 2015-12-23 | 上海修源网络科技有限公司 | Auxiliary driving method and system |
| CN105469124A (en) * | 2015-11-20 | 2016-04-06 | 厦门雅迅网络股份有限公司 | Traffic sign classification method |
| JP2017004295A (en) * | 2015-06-11 | 2017-01-05 | 株式会社ミツバ | Traffic light recognition apparatus and traffic light recognition method |
| CN106408592A (en) * | 2016-09-09 | 2017-02-15 | 南京航空航天大学 | Target tracking method based on target template updating |
| CN107527511A (en) * | 2016-06-22 | 2017-12-29 | 杭州海康威视数字技术股份有限公司 | A kind of intelligent vehicle driving based reminding method and device |
-
2019
- 2019-03-22 CN CN201910222479.6A patent/CN111723625B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20070074913A (en) * | 2006-01-11 | 2007-07-18 | 주식회사 현대오토넷 | System and method for reporting change of signal lamp using image recognition |
| KR101096806B1 (en) * | 2010-10-13 | 2011-12-22 | 한국전자통신연구원 | System for informing traffic signal |
| DE102013101175A1 (en) * | 2012-02-13 | 2013-08-14 | Toyota Motor Engineering & Manufacturing North America Inc. | System and method for detecting traffic signals |
| US20130211682A1 (en) * | 2012-02-13 | 2013-08-15 | Toyota Motor Engin. & Manufact. N.A.(TEMA) | System and method for traffic signal recognition |
| CN102800109A (en) * | 2012-07-16 | 2012-11-28 | 刘怡光 | Method for tracking locomotive signal lamp based on particle filter |
| CN102819728A (en) * | 2012-07-17 | 2012-12-12 | 中国航天科工集团第三研究院第八三五七研究所 | Traffic sign detection method based on classification template matching |
| JP2017004295A (en) * | 2015-06-11 | 2017-01-05 | 株式会社ミツバ | Traffic light recognition apparatus and traffic light recognition method |
| CN105185140A (en) * | 2015-09-30 | 2015-12-23 | 上海修源网络科技有限公司 | Auxiliary driving method and system |
| CN105469124A (en) * | 2015-11-20 | 2016-04-06 | 厦门雅迅网络股份有限公司 | Traffic sign classification method |
| CN107527511A (en) * | 2016-06-22 | 2017-12-29 | 杭州海康威视数字技术股份有限公司 | A kind of intelligent vehicle driving based reminding method and device |
| CN106408592A (en) * | 2016-09-09 | 2017-02-15 | 南京航空航天大学 | Target tracking method based on target template updating |
Non-Patent Citations (2)
| Title |
|---|
| 李维维等: "《基于特征学习与特征记忆模板更新机制的粒子滤波跟踪》", 《中国科学技术大学学报》, pages 292 - 302 * |
| 武莹;张小宁;何斌;: "基于图像处理的交通信号灯识别方法", 交通信息与安全, no. 03 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112150364A (en) * | 2020-10-10 | 2020-12-29 | 厦门理工学院 | Pairing and splicing method for split type candidate image areas of arrow-shaped traffic signal lamp |
| CN112150364B (en) * | 2020-10-10 | 2022-06-07 | 厦门理工学院 | Pairing and splicing method for split type candidate image areas of arrow-shaped traffic signal lamp |
| CN112528794A (en) * | 2020-12-03 | 2021-03-19 | 北京百度网讯科技有限公司 | Signal lamp fault identification method and device and road side equipment |
| CN113053152A (en) * | 2021-03-08 | 2021-06-29 | 江铃汽车股份有限公司 | Vehicle-mounted traffic sign real-time display method and system |
| CN113052190A (en) * | 2021-04-01 | 2021-06-29 | 威海职业学院(威海市技术学院) | Visual auxiliary system and method based on image processing |
| CN113052190B (en) * | 2021-04-01 | 2022-05-31 | 威海职业学院(威海市技术学院) | Visual auxiliary system and method based on image processing |
| CN113989774A (en) * | 2021-10-27 | 2022-01-28 | 广州小鹏自动驾驶科技有限公司 | Traffic light detection method and device, vehicle and readable storage medium |
| CN116152784A (en) * | 2023-04-21 | 2023-05-23 | 深圳市夜行人科技有限公司 | Signal lamp early warning method and system based on image processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111723625B (en) | 2023-08-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111723625B (en) | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium | |
| US11694430B2 (en) | Brake light detection | |
| US10650257B2 (en) | Method and device for identifying the signaling state of at least one signaling device | |
| CN103984950B (en) | A kind of moving vehicle brake light status recognition methods for adapting to detection on daytime | |
| Huang et al. | Vehicle detection and inter-vehicle distance estimation using single-lens video camera on urban/suburb roads | |
| KR101982298B1 (en) | Method and apparatus for detecting sudden moving objecj appearance at vehicle | |
| CN110929655B (en) | Lane line identification method in driving process, terminal device and storage medium | |
| JP2002083297A (en) | Object recognition method and object recognition device | |
| CN107886034B (en) | Driving reminding method and device and vehicle | |
| CN113297881B (en) | Target detection method and related device | |
| CN112677977B (en) | Driving state identification method and device, electronic equipment and steering lamp control method | |
| CN110458050B (en) | Vehicle cut-in detection method and device based on vehicle-mounted video | |
| CN110991264A (en) | Front vehicle detection method and device | |
| CN113255444A (en) | Training method of image recognition model, image recognition method and device | |
| CN113989772A (en) | Traffic light detection method and device, vehicle and readable storage medium | |
| CN111332306A (en) | Traffic road perception auxiliary driving early warning device based on machine vision | |
| CN114842432A (en) | Automobile light control method and system based on deep learning | |
| CN113361299A (en) | Abnormal parking detection method and device, storage medium and electronic equipment | |
| Ben Romdhane et al. | A lane detection and tracking method for driver assistance system | |
| CN113611008B (en) | Vehicle driving scene acquisition method, device, equipment and medium | |
| CN110414357B (en) | Front vehicle positioning method based on vehicle type identification | |
| CN113715827A (en) | Driving reminding method and device, electronic equipment and storage medium | |
| Hammami et al. | An improved lane detection and tracking method for lane departure warning systems | |
| CN111611942B (en) | Method for extracting and building database by perspective self-adaptive lane skeleton | |
| CN114820691B (en) | Method, device and equipment for detecting motion state of vehicle and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |