CN111723625B - Traffic light image recognition processing method and device, auxiliary traffic system and storage medium - Google Patents
Traffic light image recognition processing method and device, auxiliary traffic system and storage medium Download PDFInfo
- Publication number
- CN111723625B CN111723625B CN201910222479.6A CN201910222479A CN111723625B CN 111723625 B CN111723625 B CN 111723625B CN 201910222479 A CN201910222479 A CN 201910222479A CN 111723625 B CN111723625 B CN 111723625B
- Authority
- CN
- China
- Prior art keywords
- information
- image
- traffic light
- template
- template library
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
- G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
- G08G1/09623—Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a traffic light image recognition processing method, which comprises the following steps: s100: acquiring an image containing depth information and color information and preprocessing the image; s200: positioning a traffic light background frame in the image; s300: identifying the type and the color of the traffic lights in the background frame; s400: and carrying out target tracking on the traffic light, and outputting state information and distance information of the traffic light when the target tracking result meets the preset condition. The invention also provides a traffic light image recognition processing device, an auxiliary passing system and a computer readable storage medium. After the technical scheme is adopted, the traffic light can be accurately identified, the false alarm risk is obviously reduced, and effective assistance is provided for the passing of drivers.
Description
Technical Field
The present invention relates to the field of auxiliary traffic technologies, and in particular, to a traffic light image recognition processing method and apparatus, an auxiliary traffic system, and a storage medium.
Background
In the prior art, the intersection auxiliary passing system generally comprises a camera arranged at the center of the upper part of a front windshield of an automobile, the camera is connected with a controller in the automobile through a wire, the controller is further connected with a voice alarm through the wire, the controller acquires images of a front traffic light through the camera and recognizes the color of the traffic light, and then the voice alarm is controlled to send out corresponding voice instructions according to the recognized color.
However, in the prior art, the processing of traffic light targets is generally limited to recognition, that is, a camera collects an image including road conditions at an intersection, a vehicle-mounted embedded processor processes the collected single-frame image, and then early warning judgment is made, and early warning information is sent to a user according to a judgment result, so that the vehicle-mounted embedded processor is used as a means for assisting in driving a vehicle. However, the method has obvious false alarm risk, and has no better solution to complex situations such as traffic light color switching, rapid departure of the automobile from the intersection and the like. Most of the systems are combined with signal generating devices arranged on traffic lights at the intersections, the devices send traffic light state information to the vehicle when the vehicle passes through the intersections, and the vehicle obtains the information and then performs discrimination processing by combining with own state data such as vehicle speed and the like, so that an early warning function is realized, and the implementation complexity of a scheme is obviously increased.
Therefore, there is a need to develop a traffic light image recognition processing method, a device, an auxiliary traffic system and a storage medium capable of accurately recognizing traffic lights and reducing the risk of false alarm.
Disclosure of Invention
In order to overcome the technical defects, the invention aims to provide a traffic light image recognition processing method, a device, an auxiliary passing system and a storage medium, wherein the traffic light image recognition processing method, the device and the auxiliary passing system can accurately recognize traffic lights and reduce false alarm risks.
The invention discloses an image recognition processing method for traffic lights, which comprises the following steps:
s100: acquiring an image containing depth information and color information and preprocessing the image;
s200: positioning a traffic light background frame in the image;
s300: identifying the type and the color of the traffic lights in the background frame;
s400: and carrying out target tracking on the traffic light, and outputting state information and distance information of the traffic light when the target tracking result meets the preset condition.
Preferably, the step S400 includes:
s401: analyzing the first frame image after the steps S100-S300, extracting feature vectors, and storing the feature vectors as templates in a template library;
s402: extracting feature vectors from images in subsequent frames after the steps S100-S300, and then matching the feature vectors with a template library;
s403: updating the template library according to the matching result;
s404: and outputting the state information and the distance information of the traffic light when the number of frames successfully matched continuously exceeds the set number of frames.
Preferably, the step S401 specifically includes:
analyzing the image of the first frame to obtain the mean value, variance and distribution information of surrounding feature points of gray values in a preset range near the feature points;
establishing attribute vectors of the feature points according to the analyzed information, and storing the attribute vectors as templates in a template library;
the step S402 specifically includes:
analyzing images in subsequent frames, establishing attribute vectors of feature points, matching the attribute vectors with templates in a template library, and comparing the similarity;
the step S403 specifically includes:
when the attribute vector of the image characteristic point in the subsequent frame is completely the same as any template in the template library, covering the templates in the template library; when the attribute vector of the image characteristic point in the subsequent frame is not completely the same as the template of the template library, the image characteristic point is stored as a new template in the template library;
defining a reliability weight of the template according to the use frequency of the template, and updating the template in the template library according to the reliability weight when the new template is stored in the template library;
the step S404 specifically includes:
comparing the attribute vector of the image characteristic point in the subsequent frame with the reliability weight threshold of the template library, and judging that the matching is successful when the attribute vector of the image characteristic point in the subsequent frame is larger than the reliability weight threshold of the template library;
when the number of frames successfully matched continuously exceeds the set number of frames, outputting the state information and the distance information of the traffic lights of the current frame;
when the traffic light is a round traffic light, the state information is the color of the traffic light; when the traffic light is an arrow traffic light, the state information is the direction and the color of the arrow;
the distance information is depth information of traffic light areas on the image.
Preferably, the attribute vector is:
V＝(m 1 ，m 2 ，μ，σ，p0，p 1 ，…，p 8 ，)
m 1 to the maximum distribution of other characteristic points in a preset range around the characteristic pointIs a distance of (2); m is m 2 The distance of other characteristic points distributed for a plurality of times in a preset range around the characteristic point; mu is the gray variance of 9 pixels around the feature point, sigma is the gray standard deviation of 9 pixels around the feature point, p 0 ，p 1 ，...，p 8 The 9 pixels around the feature point are arranged from large to small in gray scale.
Preferably, the reliability weight of the template in the template library at the time t is defined as:
u (-) is a unit step function; beta is a coefficient vector which is the product of the attribute vector of the feature point and the use frequency of the template; beta i The coefficient corresponding to the ith template is obtained by a coefficient vector through a cosine theorem; n is the total number of templates;
the reliability weight threshold of the template library is the product of the average value of the reliability weight of the templates in the template library and a preset coefficient;
updating templates in a template library according to the reliability weight, specifically comprising: when the number of templates in the template library does not reach the total number N, deleting the templates with reliability weight lower than the reliability weight threshold of the template library when the new templates are stored in the template library; when the number of templates in the template library reaches the total number N and templates with similarity lower than the reliability weight threshold value of the template library exist, deleting the templates with the reliability weight lower than the reliability weight threshold value of the template library when new templates are stored in the template library; and randomly extracting and updating the templates with the reliability weight higher than the reliability weight threshold of the template library when the number of the templates in the template library reaches the total number N and the similarity of the templates is higher than the reliability weight threshold of the template library.
Preferably, the step S400 further includes the following steps:
s500: acquiring running information of a vehicle, and sending out reminding information when the running information of the vehicle, the color and the distance of an output traffic light meet preset conditions;
the travel information includes: one or more of vehicle speed information, steering lamp information, wiper information and gear information;
the reminding information comprises: one or more of voice prompt, light prompt and vibration prompt.
The invention also discloses a traffic light image recognition processing device, which comprises:
the acquisition module is used for acquiring an image containing depth information and color information and preprocessing the image;
the positioning module is used for positioning a traffic light background frame in the image;
the identification module is used for identifying the type and the color of the traffic light in the background frame;
the tracking module is used for tracking the target of the traffic light and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
The invention also discloses an auxiliary passing system, which comprises:
the image acquisition module is used for acquiring an image containing depth information and color information;
the whole vehicle communication module is communicated with the vehicle body CAN network and used for acquiring the running information of the vehicle;
the data processing module is in communication connection with the image acquisition module and the whole vehicle communication module, the data processing module comprises a processor, a memory and an I/O port, the memory is used for storing a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions and execute the image recognition processing method;
and the prompt and early warning module is communicated with the whole vehicle communication module through a vehicle body CAN network and is used for sending out prompt information.
Preferably, the image acquisition module comprises a 3D camera having a CMOS image sensor incorporating TOF;
the prompting and early warning module comprises one or more of a loudspeaker, an eccentric motor and a warning lamp.
The invention also discloses a computer readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the image recognition processing method described above.
After the technical scheme is adopted, compared with the prior art, the method has the following beneficial effects:
1. the traffic light can be accurately identified, and the false alarm risk is obviously reduced;
2. the search space in the template matching process is not obviously increased along with the increase of the image size and the template size, so that the complexity of algorithm calculation is reduced, and the efficiency of target tracking is improved.
3. And adding target reconstruction before matching a new frame of image, and predicting a possible region of the target, so that the searching range is reduced, the iteration times of tracking operation are reduced to a certain extent, and the real-time performance of the system is improved.
Drawings
FIG. 1 is a flowchart of an image recognition processing method according to an embodiment of the invention;
FIG. 2 is a flowchart of an image recognition processing method according to an embodiment of the invention
Fig. 3 is a schematic structural diagram of an auxiliary traffic system according to an embodiment of the invention.
Reference numerals:
100-image acquisition module, 120-data processing module, 130-whole car communication module, 140-prompt and early warning module.
Detailed Description
Advantages of the invention are further illustrated in the following description, taken in conjunction with the accompanying drawings and detailed description.
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
In the description of the present invention, it should be understood that the terms "longitudinal," "transverse," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate describing the present invention and simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present invention.
In the description of the present invention, unless otherwise specified and defined, it should be noted that the terms "mounted," "connected," and "coupled" are to be construed broadly, and may be, for example, mechanical or electrical, or may be in communication with each other between two elements, directly or indirectly through intermediaries, as would be understood by those skilled in the art, in view of the specific meaning of the terms described above.
In the following description, suffixes such as "module", "component", or "unit" for representing elements are used only for facilitating the description of the present invention, and are not of specific significance per se. Thus, "module" and "component" may be used in combination.
Referring to fig. 1, the invention provides a traffic light image recognition processing method, which comprises the following steps:
s100: an image containing depth information and color information is acquired and preprocessed.
Specifically, the step S100 includes:
s101: an image including depth information and color information is acquired by an image acquisition module provided on a vehicle. The image acquisition module may be a 3D camera disposed on the vehicle. Preferably, a CMOS (Complementary Metal Oxide Semiconductor) image sensor combined with TOF (Time of Flight) is provided in the 3D camera, which can obtain a depth image and a normal RGB color image at the same time, and each pixel unit of the sensor contains 1Z pixel (for generating a depth image) and 8 RGB pixels (for generating an RGB color image), and the CMOS sensor generates one RGB color image and one depth image at almost the same time when the same set of optical lenses is used.
S102: and performing histogram equalization operation on the color-based image. Through the histogram equalization operation, the color tone of the image is enhanced, thereby facilitating the recognition of the color. Preferably, before step S102, the method further comprises a step of extracting an image, wherein step S102 is performed for extracting a region about 1/3 to 1/2 of the middle of the image obtained in step S101. By such processing, the region of interest of image processing can be reduced, the amount of data processed can be reduced, and the influence of a part of other light sources (such as a part of automobile tail lights and the like) can be reduced.
S103: the image obtained in step S102 is denoised. Preferably, gaussian filtering is used for denoising.
S104: the color of the large area in the image obtained in step S103 is filtered out by morphological processing, and the color of the bright small area is retained. At this time, the obtained image completes preprocessing.
After step S100 is completed, the steps are performed:
s200: and positioning a traffic light background frame in the image.
Specifically, step S200 includes:
s201: and (3) converting the image obtained in the step (S100) into an HSV color space, and dividing the image by utilizing the characteristics of the three colors of red, green and yellow to obtain a target divided image with the characteristics of the colors of red, green and yellow.
S202: and analyzing the depth information of the image in S201, extracting a target area in a set depth range, and performing morphological processing on the target area. Because the acquired image contains depth information and color information, that is, each pixel unit in the image has a distance information, a proper screening is performed according to a preset depth threshold range, so that some interference areas can be eliminated. One possible way to set the preset depth threshold range is: according to the setting and installation specifications of road traffic signal lamps of GB 14886-2006, a target area which is too close or too far in an image is an interference area, and an area in a proper range (such as a range of 50-200 m) is an area which needs to be extracted from the target image. Meanwhile, the installation angle of the image acquisition module (such as a 3D camera) can be combined, and the area with the overlarge or overlarge angle in the image can be discharged. And carrying out morphological processing on the extracted target image area to remove noise interference in the image. For example, morphological operations are performed on the target image to remove small protrusions (e.g., connection between a traffic light background frame and a traffic light pole, etc.) in the image.
S203: and (3) performing background detection on the image obtained in the step S202, and detecting a rectangular black frame with the rectangular degree or the length-width ratio within a preset range. The value of the rectangle degree R is between 0 and 1. When the object is rectangular, R takes a maximum value of 1. A rectangle degree threshold (e.g., 0.9) may be set, that is, when the rectangle degree of the background area is greater than the set rectangle degree threshold, the background area is considered to be a rectangular background frame, and an area that does not satisfy the rectangle degree threshold may be excluded. Thus, the positioning of the traffic light background frame is realized.
After step S200 is completed, the steps are performed:
s300: and identifying the type and the color of the traffic light in the background frame.
Specifically, in step S300, the type of the traffic light is first determined according to the shape of the traffic light in the traffic light background frame. And dividing the candidate area in the background frame of the traffic light, and detecting the length-width ratio of the candidate area. The aspect ratio of a circular traffic light should ideally be 1:1, if the length and width proportions of the candidate regions in the segmented image are very different, it can be determined that the region is not a circular traffic light. Setting the length-width ratio between 0.7 and 1.4, and if the area is considered to be a circular traffic light, performing step S310; if the area is not found to be the circular traffic light, the area is recognized as an arrow traffic light, and the process proceeds to step S320.
S310: circular traffic light discernment includes the following steps:
s311: binarizing the image obtained based on the step 300 by Otsu (maximum inter-class variance method);
s312: the color image including the traffic light background frame area is converted into a Lab color space from an RGB color space, then the characteristics of three components L, a and b in the Lab color space are utilized to be segmented with the information of the image obtained in the step S311, an initial segmented image is obtained, and the color type is judged by the selection range of the traffic light colors based on a and b. By way of example, all possible light colors of a round traffic light are red, green and yellow. The threshold range of green is: -50< a < -8 and 15< b <80; the threshold range of red is 15< a <110 and 15< b <60; the threshold range of yellow is 1< a <16 and 25< b <60. So that in this step the color of the traffic light can be determined.
S313: and analyzing the depth information of the obtained image in S312, extracting a target area in a set depth range, and performing morphological processing on the target area to exclude areas which do not meet the requirements. This step is similar to step S202 and will not be described again here.
S314: and obtaining the minimum circumscribed rectangle of the region of interest by calculating the region of interest. The region of interest refers to a region to be processed outlined from the processed image in the form of a square, circle, ellipse, irregular polygon, etc. In image processing, a region of interest (Region of Interest, ROI) of an image is usually obtained by an operator and a function, the calculation range is narrowed, and then the next processing of the image is performed. This area is the focus of attention for your image analysis. The region is delineated for further processing. The target to be read is defined by using the ROI, so that the processing time can be reduced, and the precision can be increased. The region of interest herein refers to the region of the traffic light that meets the requirements of a circular traffic light in terms of aspect ratio within the frame of the traffic light.
S315: the Hu feature of the template of the circular traffic light and the image of the region of interest is obtained by using seven functions of the Hu invariant moment, then the feature values of the regions of interest are matched by using the obtained template feature values, and if the obtained values are smaller, the similarity of the obtained values is indicated. And judging a round lamp when the value of the similarity is within a preset range. The templates herein refer to templates of various types of traffic lights in a template library. The method is used for accurately identifying the round traffic lights.
S320: the arrow traffic light identification comprises the following steps:
s321: the image obtained in step S310 is converted into HSV color space. Dividing the image by utilizing the characteristics of three colors of red, green and yellow to obtain a target divided image with the characteristics of the red, green and yellow;
s322: carrying out gray-scale treatment on the partial image of the background frame of the traffic light;
s323: performing binarization processing on the obtained gray level image, performing regional automatic threshold binarization processing on the obtained gray level image by combining the color segmentation image, and separating out an arrow traffic light;
s324: performing a closing operation on the obtained binarized image;
s325: extracting a minimum arrow target area and carrying out normalization treatment;
s326: the arrow type identification is performed by utilizing the characteristics of the arrow traffic signal lamp. The characteristics of the traffic light signal itself, i.e. the symmetry of the arrows. For example, when the arrow traffic light is symmetrical about the horizontal direction symmetry axis and is not symmetrical about the vertical direction symmetry axis, determining that the arrow of the arrow traffic light points to the left or right by comparing the number of target points located on both sides of the vertical direction symmetry axis; if the number of target points (the target points are pixel points forming the arrow traffic light) positioned on the left side of the vertical direction symmetry axis is larger than the number of target points positioned on the right side of the vertical direction symmetry axis, determining that the arrow type of the arrow traffic light is straight left; and if the number of the target points positioned on the left side of the vertical direction symmetry axis is smaller than the number of the target points positioned on the right side of the vertical direction symmetry axis, determining that the arrow type of the arrow traffic light is right straight. In the step, the arrow direction information of the arrow traffic light can be determined, and the light color of the arrow traffic light can be obtained by combining the color segmentation result.
S400: and carrying out target tracking on the traffic light, and outputting state information and distance information of the traffic light when the target tracking result meets the preset condition.
Specifically, S400 includes the steps of:
s401: analyzing the first frame image after the steps S100-S300, extracting the characteristic vector, and storing the characteristic vector as a template in a template library.
The image acquisition module (for example, a 3D camera) may acquire images of consecutive frames, and steps S100 to S300 are performed. Analyzing the image of the first frame after the steps S100-S300 to obtain the mean value, variance and distribution information of surrounding feature points of gray values in a preset range near the feature points; and establishing attribute vectors of the feature points according to the analyzed information, and storing the attribute vectors as templates in a template library.
In image processing, feature points refer to points where the gray value of an image changes drastically or points where the curvature is large on the edge of an image (i.e., the intersection of two edges). The image feature points play a very important role in the feature point-based image matching algorithm. The image feature points can reflect the essential features of the image and can identify the target object in the image. Matching of images can be completed through matching of feature points.
Specifically, the attribute vector of the feature point
V＝(m 1 ，m 2 ，μ，σ，p 0 ，p 1 ，…，p 8 ，)
m 1 The distance at which other characteristic points are most distributed in a preset range around the characteristic points is set; m is m 2 The distance of other characteristic points distributed for a plurality of times in a preset range around the characteristic point; mu is the gray variance of 9 pixels around the feature point, sigma is the gray standard deviation of 9 pixels around the feature point, p 0 ，p 1 ，...，p 8 The 9 pixels around the feature point are arranged from large to small in gray scale.
S402: and extracting feature vectors from the images in the subsequent frames after the steps S100-S300, and then matching the feature vectors with a template library.
Analyzing the images in the subsequent frames passing through the steps S100-S300, establishing attribute vectors V of the feature points, and then matching with templates in a template library to compare the similarity. Further, in this step, the relation between the coordinates of the target center point and the boundary of the tracking area is analyzed, and whether the target object is unable to track due to the tendency to be stationary or stops tracking due to the target having been separated from the tracking area is determined, and if the target center point is located in the tracking area, we call template matching to search, and estimate, for each frame, an observation sequence from the start of tracking to the current one.
S403: and updating the template library according to the matching result.
Specifically, when the attribute vector of the image feature point in the subsequent frame is completely the same as any template in the template library, covering the templates in the template library; and when the attribute vector of the image characteristic point in the subsequent frame is not completely the same as the template of the template library, storing the attribute vector as a new template into the template library.
For templates in the template library, the reliability weight of the template is defined according to the use frequency of the template. The usage frequency of a template refers to the number of times a corresponding template in the template library is covered by a new template per unit time. The reliability weight of the template at the time t is defined as
Wherein U ()' is a unit step function; beta is a coefficient vector which is the product of the attribute vector of the feature point and the use frequency of the template; beta i The coefficient corresponding to the ith template is obtained by a coefficient vector through a cosine theorem; n is the total number of templates. The template coefficient is a mathematical distance and can be used to measure the similarity between two coefficient vectors, and the relationship between the template coefficient and the coefficient vector can be obtained by cosine theorem. Such as:
cos(A)＝(b 1 c 1 +b 2 c 2 +…b n c n )/sqrt[(b 1 2 +b 2 2 +…+b n 2 )(c 1 2 +c 2 2 +…+c n 2 )]
where the numerator represents the inner product of the two vectors and the denominator represents the modulo multiplication of the two vectors.
In order to ensure the diversity of templates, when a new template is stored in a template library, one template with the reliability weight higher than a set threshold value can be randomly extracted for updating, and the template with the reliability weight lower than the set threshold value can be deleted. The set threshold here refers to a reliability weight threshold of the module library. The reliability weight threshold of the template library can be obtained by multiplying the average value of the reliability weights of the templates in the template library by a preset coefficient, wherein the preset coefficient is smaller than 1, and preferably 3/5. As can be seen from the formula, the reliability weight changes after the template library is updated, and correspondingly, the threshold value of the reliability weight changes. Preferably, when the number of templates in the template library does not reach the total number N, deleting the templates with reliability weights lower than a set threshold value when new templates are stored in the template library; when the number of templates in the template library reaches the total number N and templates with similarity lower than a set threshold value exist, deleting the templates with reliability weight lower than the set threshold value when new templates are stored in the template library; and randomly extracting one template with reliability weight higher than a set threshold value to update when the number of templates in the template library reaches the total number N and the similarity of the templates is higher than the set threshold value.
S404: and outputting the state information and the distance information of the traffic light when the number of frames successfully matched continuously exceeds the set number of frames.
And comparing the attribute vector of the image characteristic point in the subsequent frame with the reliability weight threshold of the template library, and judging that the matching is successful when the attribute vector of the image characteristic point in the subsequent frame is larger than the reliability weight threshold of the template library.
When the number of frames successfully matched in succession exceeds the set number of frames, preferably, the set number of frames is 10 frames, and the state information and the distance information of the traffic light of the current frame are output. When the traffic light is a round traffic light, the state information is the color of the traffic light; when the traffic light is an arrow traffic light, the state information is the direction and the color of the arrow; the distance information is depth information of a traffic light area on the image, namely distance information from the traffic light to an image acquisition module (such as a 3D camera). The state information of the traffic light is obtained in step S300, and the distance information may be directly obtained through the depth image acquired by the image acquisition module. The state information and the distance information of the traffic light can be output to the outside or can be output to a subsequent program for processing. And when the number of frames successfully matched in succession does not meet the preset number of frames, the state information and the distance information of the traffic light are not output.
Referring to fig. 2, step S400 preferably further includes the steps of:
s500: acquiring running information of a vehicle, and sending out reminding information when the running information of the vehicle, the color and the distance of an output traffic light meet preset conditions; the travel information includes: one or more of vehicle speed information, steering lamp information, wiper information and gear information; the reminding information comprises: one or more of voice prompt, light prompt and vibration prompt.
Specifically, vehicle driving information required by intersection auxiliary judgment, namely real-time data of the whole vehicle, such as one or more of vehicle speed information, turn signal light information, wiper information and gear information, is obtained through a vehicle body CAN network. And then makes a decision in combination with the traffic light status information and the distance information output in step S400.
For example, if the car goes straight and is in a low-speed driving state, the state of the traffic light is red light (red arrow or red round light), and when the depth information of the traffic light (i.e. the distance from the car to the traffic light) is smaller than a preset low-speed alarm threshold value, a pre-alarm message signal is sent to the CAN network of the whole car, a mechanical motor arranged in the steering wheel sends out low-frequency vibration, the alarm light in the car is normally on, and a built-in loudspeaker plays alarm voice, so that the driver is reminded.
If the automobile is in a straight running state and is in a high-speed running state, the state of a traffic light is a red light (red arrow or red round light), and when the depth information of the traffic light (namely the distance from the automobile to the traffic light) is smaller than a preset high-speed alarm threshold value, a pre-alarm message signal is sent to a CAN (controller area network) of the whole automobile, a control instruction is sent by the whole automobile, a mechanical motor arranged in a steering wheel sends out high-frequency vibration, and the alarm light flashes; the built-in speaker plays the alarm voice to remind the driver.
If the automobile turns on the left turn light, the state of the traffic light is 10 frames or more of red light (red left turn arrow or red round light), and when the depth information of the traffic light (namely the distance from the automobile to the traffic light) is smaller than a preset low-speed alarm threshold value, a pre-alarm message signal is sent to the CAN network of the whole automobile, a mechanical motor arranged in the steering wheel sends out high-frequency vibration, and an alarm voice prompt is generated in the automobile.
If the automobile goes straight, the traffic light is in a yellow light (red arrow or round light) flashing state, and then a pre-alarm message signal is sent to the CAN network of the whole automobile in a micro-mode, and the alarm light flashes.
Preferably, when the windscreen wiper is opened for more than the preset time, the weather is considered to be rainy, and the value of the alarm threshold is appropriately adjusted to adapt to the weather.
The above only are several preferred judging rules and reminding modes, and can be flexibly set according to the needs.
According to the technical scheme, the traffic light is subjected to target tracking, and after the traffic light image is successfully matched with the templates in the template library by the preset frame number, the traffic state information and the distance information are output for subsequent processing, so that the accuracy of traffic light identification is improved, and the risk of false alarm is remarkably reduced. Meanwhile, as the traffic lights in the traffic light background frame are subjected to target tracking, the search space in the template matching process is not obviously increased along with the increase of the image size and the template size, the complexity of algorithm calculation is reduced, and the target tracking efficiency is improved. Meanwhile, a target reconstruction is added before a new frame of image is matched, prediction is made on a possible region of the target, the searching range is reduced, the iteration times of tracking operation are reduced to a certain extent, and the instantaneity of the system is improved.
The invention also provides a traffic light image recognition processing device, which comprises:
the acquisition module is used for acquiring an image containing depth information and color information and preprocessing the image;
the positioning module is used for positioning a traffic light background frame in the image;
the identification module is used for identifying the type and the color of the traffic light in the background frame;
the tracking module is used for tracking the target of the traffic light and outputting the state information and the distance information of the traffic light when the target tracking result meets the preset condition.
Preferably, the image recognition processing device further includes:
the judgment reminding module is used for acquiring the running information of the vehicle and sending out reminding information when the running information of the vehicle, the color and the distance of the output traffic lights meet preset conditions.
Referring to fig. 3, the present invention further provides an auxiliary traffic system, including:
image acquisition module 110
For acquiring an image containing depth information and color information. The image acquisition module 110 preferably comprises a 3D camera. The 3D camera has a CMOS (Complementary Metal Oxide Semiconductor) image sensor in combination with TOF (Time of Flight), which can obtain a depth image and a normal RGB color image at the same time, and each pixel unit of the sensor contains 1Z pixel (for generating a depth image) and 8 RGB pixels (for generating an RGB color image), and the CMOS sensor generates one RGB color image and one depth image almost at the same time when the same set of optical lenses is used. The image acquisition module 110 is installed on the vehicle, and is used for capturing images of real-time road conditions of the road in front of the vehicle.
Whole vehicle communication module 130
The vehicle body CAN network is communicated with the vehicle body CAN network and is used for acquiring the running information of the vehicle;
data processing module 120
The data processing module 120 includes a processor, a memory, and an I/O port, where the memory is configured to store a computer program, and the computer program includes program instructions, and the processor is configured to invoke the program instructions to execute the image recognition processing method described above. Specifically, the processor is an embedded microprocessor, and the memory comprises a read-only memory for completing power-on self-check of the system, initialization of each functional module, operation and guidance of driving of input/output of the system, and a random access memory capable of being read and written at any time and used for storing temporary data in an operation program. The computer program is stored in read-only memory. The I/O port can convert information formats, set interrupt and DMA (Direct Memory Access) control logic, and realize information exchange between DSP (Digital Signal Processing) and the peripheral in an interrupt mode.
Prompt pre-warning module 140
And the vehicle communication module 130 communicates with the vehicle body CAN network to send out reminding information. Specifically, the alert and pre-warning module 140 includes one or more of a speaker, an eccentric motor, and a warning light. The loudspeaker is used for playing voice so as to remind a driver; the eccentric motor is arranged in the steering wheel and used for generating vibration to remind a driver; the warning lamp is used for reminding the driver in a flashing or normally-on mode. The data processing module 120 sends out a pre-alarm message signal to the whole vehicle CAN network (CAN bus) through the whole vehicle communication module 130; the prompt and early-warning module 140 receives the early-warning message signal through the vehicle body CAN network and sends out a corresponding prompt according to the early-warning message signal.
The present invention also provides a computer-readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the image recognition processing method described above. The computer readable storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.
It should be noted that the embodiments of the present invention are preferred and not limited in any way, and any person skilled in the art may make use of the above-disclosed technical content to change or modify the same into equivalent effective embodiments without departing from the technical scope of the present invention, and any modification or equivalent change and modification of the above-described embodiments according to the technical substance of the present invention still falls within the scope of the technical scope of the present invention.
Claims (7)
1. A traffic light image recognition processing method comprises the following steps:
s100: acquiring an image containing depth information and color information and preprocessing the image;
s200: positioning a traffic light background frame in the image;
s300: identifying the type and the color of the traffic lights in the background frame;
it is characterized by further comprising the following steps,
s400: carrying out target tracking on the traffic light, and outputting state information and distance information of the traffic light when the target tracking result meets preset conditions; comprising the following steps:
s401: analyzing the first frame of image after the steps S100-S300 to obtain the mean value, variance and distribution information of surrounding feature points of gray values in a preset range near the feature points;
establishing attribute vectors of the feature points according to the analyzed information, and storing the attribute vectors as templates in a template library;
s402: analyzing the images in the subsequent frames after the steps S100-S300, establishing attribute vectors of the feature points, matching the attribute vectors with templates in a template library, and comparing the similarity;
s403: when the attribute vector of the image characteristic point in the subsequent frame is completely the same as any template in the template library, covering the templates in the template library; when the attribute vector of the image characteristic point in the subsequent frame is not completely the same as the template of the template library, the image characteristic point is stored as a new template in the template library;
defining a reliability weight of the template according to the use frequency of the template, and updating the template in the template library according to the reliability weight when the new template is stored in the template library;
the reliability weight of the template in the template library at the time t is defined as:
u (-) is a unit step function; beta is a coefficient vector which is the product of the attribute vector of the feature point and the use frequency of the template; beta i The coefficient corresponding to the ith template is obtained by a coefficient vector through a cosine theorem; n is the total number of templates;
updating templates in a template library according to the reliability weight, specifically comprising: when the number of templates in the template library does not reach the total number N, deleting the templates with reliability weight lower than the reliability weight threshold of the template library when the new templates are stored in the template library; when the number of templates in the template library reaches the total number N and templates with similarity lower than the reliability weight threshold value of the template library exist, deleting the templates with the reliability weight lower than the reliability weight threshold value of the template library when new templates are stored in the template library; randomly extracting and updating templates with reliability weight higher than the reliability weight threshold of the template library when the number of templates in the template library reaches the total number N and the similarity of the templates is higher than the reliability weight threshold of the template library;
s404: comparing the attribute vector of the image characteristic point in the subsequent frame with the reliability weight threshold of the template library, and judging that the matching is successful when the attribute vector of the image characteristic point in the subsequent frame is larger than the reliability weight threshold of the template library;
when the number of frames successfully matched continuously exceeds the set number of frames, outputting the state information and the distance information of the traffic lights of the current frame;
when the traffic light is a round traffic light, the state information is the color of the traffic light; when the traffic light is an arrow traffic light, the state information is the direction and the color of the arrow;
the distance information is depth information of traffic light areas on the image.
2. The image recognition processing method according to claim 1, wherein,
the attribute vector is:
V＝(m 1 ，m 2 ，μ，σ，p 0 ，p 1 ，...，p 8 )
m 1 the distance at which other characteristic points are most distributed in a preset range around the characteristic points is set; m is m 2 The distance of other characteristic points distributed for a plurality of times in a preset range around the characteristic point; mu is the gray variance of 9 pixels around the feature point, sigma is the gray standard deviation of 9 pixels around the feature point, p 0 ，p 1 ，...，p 8 The 9 pixels around the feature point are arranged from large to small in gray scale.
3. The image recognition processing method according to claim 1, wherein,
the reliability weight threshold of the template library is the product of the average value of the reliability weight of the templates in the template library and a preset coefficient.
4. The image recognition processing method according to claim 1, wherein,
the step S400 further includes the following steps:
s500: acquiring running information of a vehicle, and sending out reminding information when the running information of the vehicle, the color and the distance of an output traffic light meet preset conditions;
the travel information includes: one or more of vehicle speed information, steering lamp information, wiper information and gear information;
the reminding information comprises: one or more of voice prompt, light prompt and vibration prompt.
5. An auxiliary traffic system, comprising:
the image acquisition module is used for acquiring an image containing depth information and color information;
the whole vehicle communication module is communicated with the vehicle body CAN network and used for acquiring the running information of the vehicle;
a data processing module in communication with the image acquisition module, the whole vehicle communication module, the data processing module comprising a processor, a memory and an I/O port, the memory for storing a computer program, the computer program comprising program instructions, the processor configured to invoke the program instructions to perform the method of any of claims 1-4;
and the prompt and early warning module is communicated with the whole vehicle communication module through a vehicle body CAN network and is used for sending out prompt information.
6. The auxiliary traffic system of claim 5, wherein,
the image acquisition module comprises a 3D camera, wherein the 3D camera is provided with a CMOS image sensor combined with TOF;
the prompting and early warning module comprises one or more of a loudspeaker, an eccentric motor and a warning lamp.
7. A computer-readable storage medium comprising,
the computer storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910222479.6A CN111723625B (en) | 2019-03-22 | 2019-03-22 | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910222479.6A CN111723625B (en) | 2019-03-22 | 2019-03-22 | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111723625A CN111723625A (en) | 2020-09-29 |
| CN111723625B true CN111723625B (en) | 2023-08-08 |
Family
ID=72562802
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910222479.6A Active CN111723625B (en) | 2019-03-22 | 2019-03-22 | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111723625B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112150364B (en) * | 2020-10-10 | 2022-06-07 | 厦门理工学院 | Pairing and splicing method for split type candidate image areas of arrow-shaped traffic signal lamp |
| CN112528794A (en) * | 2020-12-03 | 2021-03-19 | 北京百度网讯科技有限公司 | Signal lamp fault identification method and device and road side equipment |
| CN113053152A (en) * | 2021-03-08 | 2021-06-29 | 江铃汽车股份有限公司 | Vehicle-mounted traffic sign real-time display method and system |
| CN113052190B (en) * | 2021-04-01 | 2022-05-31 | 威海职业学院(威海市技术学院) | Visual auxiliary system and method based on image processing |
| CN113989774A (en) * | 2021-10-27 | 2022-01-28 | 广州小鹏自动驾驶科技有限公司 | Traffic light detection method and device, vehicle and readable storage medium |
| CN116152784B (en) * | 2023-04-21 | 2023-07-07 | 深圳市夜行人科技有限公司 | Signal lamp early warning method and system based on image processing |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20070074913A (en) * | 2006-01-11 | 2007-07-18 | 주식회사 현대오토넷 | System and method for reporting change of signal lamp using image recognition |
| KR101096806B1 (en) * | 2010-10-13 | 2011-12-22 | 한국전자통신연구원 | System for informing traffic signal |
| CN102800109A (en) * | 2012-07-16 | 2012-11-28 | 刘怡光 | Method for tracking locomotive signal lamp based on particle filter |
| CN102819728A (en) * | 2012-07-17 | 2012-12-12 | 中国航天科工集团第三研究院第八三五七研究所 | Traffic sign detection method based on classification template matching |
| DE102013101175A1 (en) * | 2012-02-13 | 2013-08-14 | Toyota Motor Engineering & Manufacturing North America Inc. | System and method for detecting traffic signals |
| CN105185140A (en) * | 2015-09-30 | 2015-12-23 | 上海修源网络科技有限公司 | Auxiliary driving method and system |
| CN105469124A (en) * | 2015-11-20 | 2016-04-06 | 厦门雅迅网络股份有限公司 | Traffic sign classification method |
| JP2017004295A (en) * | 2015-06-11 | 2017-01-05 | 株式会社ミツバ | Traffic light recognition apparatus and traffic light recognition method |
| CN106408592A (en) * | 2016-09-09 | 2017-02-15 | 南京航空航天大学 | Target tracking method based on target template updating |
| CN107527511A (en) * | 2016-06-22 | 2017-12-29 | 杭州海康威视数字技术股份有限公司 | A kind of intelligent vehicle driving based reminding method and device |
-
2019
- 2019-03-22 CN CN201910222479.6A patent/CN111723625B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20070074913A (en) * | 2006-01-11 | 2007-07-18 | 주식회사 현대오토넷 | System and method for reporting change of signal lamp using image recognition |
| KR101096806B1 (en) * | 2010-10-13 | 2011-12-22 | 한국전자통신연구원 | System for informing traffic signal |
| DE102013101175A1 (en) * | 2012-02-13 | 2013-08-14 | Toyota Motor Engineering & Manufacturing North America Inc. | System and method for detecting traffic signals |
| CN102800109A (en) * | 2012-07-16 | 2012-11-28 | 刘怡光 | Method for tracking locomotive signal lamp based on particle filter |
| CN102819728A (en) * | 2012-07-17 | 2012-12-12 | 中国航天科工集团第三研究院第八三五七研究所 | Traffic sign detection method based on classification template matching |
| JP2017004295A (en) * | 2015-06-11 | 2017-01-05 | 株式会社ミツバ | Traffic light recognition apparatus and traffic light recognition method |
| CN105185140A (en) * | 2015-09-30 | 2015-12-23 | 上海修源网络科技有限公司 | Auxiliary driving method and system |
| CN105469124A (en) * | 2015-11-20 | 2016-04-06 | 厦门雅迅网络股份有限公司 | Traffic sign classification method |
| CN107527511A (en) * | 2016-06-22 | 2017-12-29 | 杭州海康威视数字技术股份有限公司 | A kind of intelligent vehicle driving based reminding method and device |
| CN106408592A (en) * | 2016-09-09 | 2017-02-15 | 南京航空航天大学 | Target tracking method based on target template updating |
Non-Patent Citations (1)
| Title |
|---|
| 基于图像处理的交通信号灯识别方法;武莹;张小宁;何斌;;交通信息与安全(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111723625A (en) | 2020-09-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111723625B (en) | Traffic light image recognition processing method and device, auxiliary traffic system and storage medium | |
| US11694430B2 (en) | Brake light detection | |
| CN109506664B (en) | Guide information providing device and method using pedestrian crossing recognition result | |
| US8184159B2 (en) | Forward looking sensor system | |
| CN109190523B (en) | Vehicle detection tracking early warning method based on vision | |
| CN107891808B (en) | Driving reminding method and device and vehicle | |
| KR101772178B1 (en) | Land mark detecting apparatus and land mark detection method for vehicle | |
| CN107886034B (en) | Driving reminding method and device and vehicle | |
| JP2002083297A (en) | Object recognition method and object recognition device | |
| KR101240499B1 (en) | Device and method for real time lane recogniton and car detection | |
| CN112677977B (en) | Driving state identification method and device, electronic equipment and steering lamp control method | |
| CN112750170B (en) | Fog feature recognition method and device and related equipment | |
| CN113297881B (en) | Target detection method and related device | |
| CN110682907B (en) | Automobile rear-end collision prevention control system and method | |
| JP5077088B2 (en) | Image processing apparatus and image processing method | |
| CN107886033B (en) | Method and device for identifying circular traffic light and vehicle | |
| CN111505617B (en) | Vehicle positioning method, device, equipment and storage medium | |
| CN107506739B (en) | Night forward vehicle detection and distance measurement method | |
| CN111332306A (en) | Traffic road perception auxiliary driving early warning device based on machine vision | |
| CN114842432A (en) | Automobile light control method and system based on deep learning | |
| CN110414357B (en) | Front vehicle positioning method based on vehicle type identification | |
| CN113581196A (en) | Vehicle driving early warning method and device, computer equipment and storage medium | |
| CN112699825A (en) | Lane line identification method and device | |
| CN110556024A (en) | Anti-collision auxiliary driving method and system and computer readable storage medium | |
| CN111539243B (en) | Continuous improved automatic data collection in detection of objects of interest |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |