CN110532903A

CN110532903A - A kind of method and apparatus of traffic lights image procossing

Info

Publication number: CN110532903A
Application number: CN201910741204.3A
Authority: CN
Inventors: 庄明磊
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2019-08-12
Filing date: 2019-08-12
Publication date: 2019-12-03
Anticipated expiration: 2039-08-12
Also published as: CN110532903B

Abstract

The invention discloses a kind of method and apparatus of traffic lights image procossing, the detection accuracy to solve existing traffic light status recognizer is lower, in fact it could happen that accidentally painting etc. is problem for traffic lights position.The embodiment of the present invention determines first position region and the color state of each traffic lights in the first rectangle frame by neural network model, if the color state of the traffic lights is red, corresponding second position region is compared when being then red light with the color state of the traffic lights in preceding N frame traffic lights image by the corresponding first position region of the traffic lights that neural network model detects, if the distance in first position region and second position region is less than the second preset threshold, pixel to be processed is determined by first position region；Otherwise, pixel to be processed is determined by second position region, the position of pixel to be processed is determined by various dimensions information, so as to improve the precision of image procossing, the errors present information for avoiding passing through single neural network model output causes image accidentally to handle.

Description

A kind of method and apparatus of traffic lights image procossing

Technical field

The present invention relates to technical field of image processing, in particular to a kind of method and apparatus of traffic lights image procossing.

Background technique

The alert camera of electricity at present in traffic administration is widely used in crossing vehicle violation behavior candid photograph, punishes for break in traffic rules and regulations Penalty foundation is provided.It mainly passes through record violation vehicle behavior and traffic lights color state information generates evidence figure violating the regulations, Make to punish the reasonable of vehicle violation change, punishment is avoided to dispute on.

Under different illumination intensity, the imaging of traffic lights color is not necessarily red, green, yellow three kinds intuitive in the alert camera of electricity Color.For example, under the scenes such as environment light is insufficient, environment light is too strong, traffic lights will appear red light center color it is partially yellow or partially white, Color partially white lamp phenomenon in green light amber light center is easy to cause in evidence figure violating the regulations and occurs that red light is not red, green light is not green, amber light is not yellow The phenomenon that, so that break in traffic rules and regulations punishment behavior is lacked persuasion.Therefore, traffic lights color in the alert camera of electricity strengthen and be tinted It is the indispensable Xiang Gongneng of the alert camera of electricity.

It is existing, actual color state and the position of traffic lights can be detected by traffic light status recognizer, and to inspection The state measured carries out applying red operation for the position of red light, however, the detection accuracy of existing traffic light status recognizer is lower, The state of the traffic lights detected and position are wrong, the problems such as accidentally applying thus occur, and need to adjust in debugging process Parameter is complicated, and it is single to be applicable in scene.

In conclusion the detection accuracy of existing traffic light status recognizer is lower, in fact it could happen that traffic lights position accidentally applies.

Summary of the invention

The present invention provides a kind of method and apparatus of traffic lights image procossing, calculates to solve existing traffic light status identification The detection accuracy of method is lower, in fact it could happen that the problem of traffic lights color accidentally applies.

In a first aspect, a kind of method of traffic lights image procossing provided in an embodiment of the present invention, comprising:

Include the first rectangle frame of the traffic lights in present frame traffic lights image for any one, passes through neural network mould Type determines first position region of each traffic lights for including in first rectangle frame in first rectangle frame and acquisition institute State the color state of traffic lights when traffic lights image；

Determine neural network model output color state be red light traffic lights in preceding N frame traffic lights image face Color state corresponding second position region when being red light；

The first position region of the same traffic lights is compared with the second position region；

If it is default that the first position region of the traffic lights is no more than second at a distance from the second position region Threshold value, it is determined that the first position region is the target position region in present frame traffic lights image comprising pixel to be processed；

If the first position region of the traffic lights is at a distance from the second position region more than the second default threshold Value, it is determined that the second position region is the target position region in present frame traffic lights image comprising pixel to be processed.

The above method filters out roughly the position of each traffic lights in present frame traffic lights image, root by the second rectangle frame Corresponding first rectangle frame of present frame traffic lights image is determined according to corresponding second rectangle frame of each traffic lights, then passes through neural network Model determine each traffic lights in the first rectangle frame exact position (i.e. first position region) and corresponding color state, the face Color state is not the color that the traffic lights is shown in present frame traffic lights image but the friendship when shooting the frame traffic lights image The actual color of logical lamp, if it is determined that the traffic lights is red, then the traffic lights detected neural network model is corresponding First position region and the color state of the traffic lights in preceding N frame traffic lights image corresponding second position region when being red light It is compared, if the range difference in first position region and second position region away from excessive, i.e., more than the second preset threshold, then passes through Second position region determines pixel to be processed, if gap be no more than the second preset threshold, by first position region determine to Pixel is handled, so as to improve the precision of image procossing, image caused by the errors present information for avoiding neural network model from exporting Accidentally handle.

In a kind of optional embodiment, first rectangle frame includes at least one second rectangle frame, the same institute The distance between second rectangle frame of any two that the first rectangle frame includes is stated less than the first preset threshold, second rectangle frame Including the traffic lights in traffic lights image described at least one.

In a kind of optional embodiment, first rectangle frame is determined in the following manner:

According in the traffic lights image on the second rectangle frame of any two same position a bit, determine two the second squares The distance between shape frame；

The second rectangle frame where distance to be no more than to the point of the first preset threshold is divided into same first rectangle frame； Wherein, the second rectangle frame is located in first rectangle frame.

In a kind of optional embodiment, described determined in first rectangle frame by neural network model includes First position region of each traffic lights in first rectangle frame, comprising:

According to the central point in the corresponding first position region of each traffic lights of neural network model output described the The dimension information of coordinate and the first position region in one rectangle frame determines the traffic lights in first rectangle frame In first position region.

In a kind of optional embodiment, the color state of the determination neural network model output is red light Traffic lights corresponding second position region when color state is red light in preceding N frame traffic lights image, comprising:

The color state to neural network model output is that the traffic lights of red light is located at preceding N frame traffic lights image Middle color state is weighted in corresponding first position region when being red light, obtains the corresponding second position of the traffic lights Region.

In a kind of optional embodiment, the determination first position region is the target comprising pixel to be processed After the band of position, further includes:

It is to be processed that the brightness for determining the pixel for including in the target position region, which is more than the pixel of third predetermined threshold value, Pixel；

The HS value of the pixel to be processed is adjusted, so that color adjusted is shown as what the neural network model determined Acquire the actual color state of the traffic lights when traffic lights image.

In a kind of optional embodiment, each pixel or described that the target position includes is adjusted in the following manner The HS value of target pixel points to be processed:

Determine the average brightness value of the pixel to be processed；

The corresponding color correction parameters of the pixel to be processed are determined according to the determining average brightness value；

If the color correction parameters are not 0, the H value for adjusting the pixel to be processed is default H value；And described in judging Whether the S value of pixel to be processed is less than the color correction parameters；

If it is determined that the S value of the pixel to be processed is less than the color correction parameters, then by the S value of the pixel to be processed It is adjusted to m1 times of current S value, and returns to execution and judges whether the S value of the pixel to be processed is less than the color correction parameters The step of, until the S value of the pixel to be processed not less than the color correction parameters or reaches the adjustment pixel to be processed S value preset times；

If it is determined that the S value of the pixel to be processed is not less than the color correction parameters, then by the S of the pixel to be processed Value is adjusted to m2 times of current S value.

Second aspect, the embodiment of the invention also provides a kind of equipment of traffic lights image procossing, which includes: processing Device and memory, wherein the memory is stored with program code, when one or more computers of memory storage When program is executed by the processor, so that the equipment executes following process:

In one possible implementation, first rectangle frame includes at least one second rectangle frame, the same institute The distance between second rectangle frame of any two that the first rectangle frame includes is stated less than the first preset threshold, second rectangle frame Including the traffic lights in traffic lights image described at least one.

In one possible implementation, the processor determines first rectangle frame in the following manner:

In one possible implementation, the processor is specifically used for:

In one possible implementation, the processor is also used to:

In one possible implementation, what the processor adjusted that the target position includes in the following manner is each The HS value of pixel or the target pixel points to be processed:

Determine the average brightness value of the pixel to be processed；

The third aspect, the embodiment of the present invention also provide a kind of equipment of traffic lights image procossing, which includes:

First determining module: for for any one the first rectangle comprising the traffic lights in present frame traffic lights image Frame determines first of each traffic lights for including in first rectangle frame in first rectangle frame by neural network model The band of position and the color state for acquiring the traffic lights when traffic lights image；

Second determining module: for determining that the color state of the neural network model output is the traffic lights of red light preceding Corresponding second position region when color state is red light in N frame traffic lights image；

Comparison module: for comparing the first position region of the same traffic lights with the second position region It is right；If the first position region of the traffic lights is no more than the second preset threshold at a distance from the second position region, Then determine that the first position region is the target position region in present frame traffic lights image comprising pixel to be processed；If described The first position region of traffic lights is at a distance from the second position region more than the second preset threshold, it is determined that described Two bands of position are the target position region in present frame traffic lights image comprising pixel to be processed.

Fourth aspect, the application also provide a kind of computer storage medium, are stored thereon with computer program, the program quilt The step of first aspect the method is realized when processor executes.

In addition, second aspect technical effect brought by any implementation into fourth aspect can be found in first party Technical effect brought by different implementations in face, details are not described herein again.

Detailed description of the invention

To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment Attached drawing is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the invention, for this For the those of ordinary skill in field, without any creative labor, it can also be obtained according to these attached drawings His attached drawing.

Fig. 1 is a kind of method schematic diagram of traffic lights image procossing provided in an embodiment of the present invention；

Fig. 2 is the schematic diagram of collected traffic lights image provided in an embodiment of the present invention；

Fig. 3 is provided in an embodiment of the present invention be the traffic lights candidate frame of each traffic lights configuration in traffic lights image signal Figure；

Fig. 4 is traffic lights image corresponding first rectangle frame provided in an embodiment of the present invention determined according to clustering algorithm Schematic diagram；

Fig. 5 is a kind of color state and position that traffic lights is detected by neural network model provided in an embodiment of the present invention Method related procedure operation chart；

Fig. 6 is a kind of processing flow schematic diagram of neural network model detection process provided in an embodiment of the present invention；

Fig. 7 is a kind of location information signal of traffic lights that neural network model detects provided in an embodiment of the present invention Figure；

Fig. 8 is a kind of method flow signal of the corresponding second location information of determining traffic lights provided in an embodiment of the present invention Figure；

Fig. 9 is a kind of schematic diagram of a scenario of the corresponding second location information of determining traffic lights provided in an embodiment of the present invention；

Figure 10 is the schematic diagram of a scenario that a kind of determining traffic lamp group provided in an embodiment of the present invention corresponds to first position region；

Figure 11 is that a kind of traffic lamp group of determining band interference provided in an embodiment of the present invention corresponds to the field in first position region Scape schematic diagram；

Scene comparison signal when Figure 12 is a kind of traffic lamp group green light provided in an embodiment of the present invention and when red light is bright Figure；

Figure 13 is the schematic diagram of a scenario of the chaff interferent of straight trip red light in removal traffic lamp group provided in an embodiment of the present invention；

Figure 14 is the structural schematic diagram of the equipment of the first traffic lights image procossing of the embodiment of the present invention；

Figure 15 is the structural schematic diagram of the equipment of second of traffic lights image procossing of the embodiment of the present invention.

Specific embodiment

To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with attached drawing to the present invention make into It is described in detail to one step, it is clear that the described embodiments are only some of the embodiments of the present invention, rather than whole implementation Example.Based on the embodiments of the present invention, obtained by those of ordinary skill in the art without making creative efforts All other embodiment, shall fall within the protection scope of the present invention.

The some words occurred in text are explained below:

1, term "and/or" in the embodiment of the present invention describes the incidence relation of affiliated partner, indicates that there may be three kinds of passes System, for example, A and/or B, can indicate: individualism A exists simultaneously A and B, these three situations of individualism B.Character "/" one As indicate forward-backward correlation object be a kind of "or" relationship.

2, term " multiple " refers to two or more in the embodiment of the present application, and other quantifiers are similar therewith.

The application scenarios of description of the embodiment of the present invention are the technical solutions in order to more clearly illustrate the embodiment of the present invention, The restriction for technical solution provided in an embodiment of the present invention is not constituted, those of ordinary skill in the art are it is found that with newly answering With the appearance of scene, technical solution provided in an embodiment of the present invention is equally applicable for similar technical problem.Wherein, at this In the description of invention, unless otherwise indicated, the meaning of " plurality " is two or more.

Currently, in traffic monitoring scene, existing high-definition digital video camera, in order to clearly distinguish type of vehicle, vehicle The information such as body color, license plate number, interior face, when needing to increase the exposure of video camera when acquiring video flowing and capturing image Between, the increase of time for exposure will lead to the pixel supersaturation in traffic lights region so that capture red light color in image it is partially yellow or Person is partially white.If red light color is shown as yellow or white, it is impossible to as to red light violation behavior punished according to According to, it is therefore desirable to the color state of traffic lights in image is captured in detection, is determined that traffic lights actual color is red light if detecting, is captured The non-red of the color that the traffic lights is shown in image then needs the color of the traffic lights being revised as red, could be used as to rushing The foundation that red light behavior is punished.

It is existing, the detection to traffic lights actual color in image is captured may be implemented by traffic light status recognizer, Red operation is applied by force to the position of the red light detected, since traffic monitoring scene is extremely complex, lamp Vitrea eye has around domain Various interference regions, such as: car light, body color etc. influence, these interference sources easily cause interference, lead to traffic light status There are some accidentally paintings or occurs phenomena such as applying wrong position, directly affects in the position inaccuracy for the red light that recognizer detects Capture the reliability of image.

The embodiment of the present invention detects the actual color state of traffic lights and the position area at place by neural network model Domain, the band of position determined when capturing the traffic lamp position red light in image by preceding N frame is weighted fusion, if neural network mould Type detect the traffic lights actual color state be red light, then by neural network model output the red light the band of position with The reference position that preceding N frame captures the traffic lights that image determines is compared, if difference is more than preset threshold, illustrates nerve net Network model has error to the position detection for currently grabbing red light in fearness image, can capture the red light that image determines according to preceding N frame Position to present frame capture image corresponding position carry out image procossing can be direct if difference is less than preset threshold Image is captured to present frame and carries out image procossing in the position of the red light detected according to neural network model.So as to improve position inspection The precision of survey reduces the number of mistake full-filling.

The embodiment of the present invention is described in further detail with reference to the accompanying drawings of the specification.

As shown in Figure 1, provided in an embodiment of the present invention is a kind of method of traffic lights image procossing, following step is specifically included It is rapid:

Step 100: for the first rectangle frame for comprising any one in present frame traffic lights image including traffic lights, leading to It crosses neural network model and determines first position of each traffic lights for including in first rectangle frame in first rectangle frame Region and the color state for acquiring the traffic lights when traffic lights image；Wherein, first rectangle frame includes at least one A second rectangle frame, the distance between second rectangle frame of any two that same first rectangle frame includes are pre- less than first If threshold value, second rectangle frame includes the traffic lights at least one traffic lights image.

Step 101: the color state for determining the neural network model output is the traffic lights of red light in preceding N frame traffic lights Corresponding second position region when color state is red light in image；

Step 102: the first position region of the same traffic lights is compared with the second position region；If The first position region of the traffic lights is no more than the second preset threshold at a distance from the second position region, it is determined that The first position region is the target position region in present frame traffic lights image comprising pixel to be processed；If the traffic lights The first position region at a distance from the second position region more than the second preset threshold, it is determined that the second position Region is the target position region in present frame traffic lights image comprising pixel to be processed.

Through the above scheme, the position of each traffic lights in present frame traffic lights image is filtered out roughly by the second rectangle frame It sets, corresponding first rectangle frame of present frame traffic lights image is determined according to corresponding second rectangle frame of each traffic lights, then pass through mind Through network model determine each traffic lights in the first rectangle frame exact position (i.e. first position region) and corresponding color shape State, the color state are not the colors that the traffic lights is shown in present frame traffic lights image but are shooting the frame traffic lights image When the traffic lights actual color, if it is determined that the traffic lights be red, then the traffic detected neural network model The corresponding first position region of lamp and the color state of the traffic lights in preceding N frame traffic lights image corresponding second when being red light Region is set to be compared, if the range difference in first position region and second position region away from excessive, i.e., more than the second preset threshold, Pixel to be processed is then determined by second position region, if gap is no more than the second preset threshold, passes through first position region Determine pixel to be processed, so as to improve the precision of image procossing, the errors present information for avoiding neural network model from exporting causes Image accidentally handle.

The embodiment of the present invention can be introduced by following three aspects, first aspect: determine present frame traffic lights image In the first rectangle frame；Second aspect: the color of the traffic lights in present frame traffic lights image is detected by neural network model State and positioned at the position of the first rectangle frame；The third aspect: position is carried out to the traffic lights that the color state detected is red light Judgement；Fourth aspect: color correction is carried out to pixel to be processed.

First aspect: the first rectangle frame in present frame traffic lights image is determined；

As shown in Fig. 2, for the traffic lights image acquired by Traffic Camera, wherein each round expression traffic lights, it is assumed that each Traffic lights all has there are three types of state, i.e., same traffic lights can show three kinds of states of red, yellow, and green.It is illustrated in figure 3 outside traffic lights The rectangle frame on side is traffic lights candidate frame, i.e., the second rectangle frame in the embodiment of the present invention.Wherein, comprising extremely in the second rectangle frame Explanation is introduced to the concrete mode for determining the first rectangle frame below for comprising a traffic lights in a few traffic lights:

Before carrying out position detection and state recognition to present frame traffic lights image, outlined roughly by traffic lights candidate frame The position of traffic lights, in general, the traffic lights of each Traffic Camera shooting fixed intersection, therefore handed in the traffic lights image shot The position of logical lamp is essentially identical, therefore, can in the traffic lights image that the camera acquires is arranged when the candidate region of each traffic lights To be configured by the web interface, such as: it pulls rectangle frame and covers entire traffic lights region.Due to monitoring camera exist shake and It rocks, may result in traffic lights candidate frame and be detached from traffic lights region, therefore, traffic lights candidate frame is greater than actual traffic lights Region, refering to Fig. 3, frame 1, frame 2, frame 3, frame 4 and frame 5 are all traffic lights candidate frame (i.e. the second rectangle frame), and are greater than respective Traffic lights region.

Traffic lights candidate frame is divided by clustering algorithm, obtains the first rectangle frame of N number of default specification, i.e., first Rectangle frame is the rectangle of identical length and width value, includes at least one traffic lights candidate frame in the first rectangle frame, and a traffic lights is waited Frame is selected to can be only present in first rectangle frame.Below traffic lights candidate frame is divided to obtain N to by clustering algorithm The process of a first rectangle frame mainly comprises the steps that

Step 1: inputting the information of each traffic lights candidate frame, index value ID and traffic lights including traffic lights candidate frame The position coordinates of candidate frame and the number of traffic lights candidate frame；Wherein, the position coordinates of traffic lights candidate frame can use (x_ Top, y_top, x_bot, y_bot) it indicates, (x_top, y_top) is the coordinate in the upper left corner of traffic lights candidate frame, (x_ Bot, y_bot) be traffic lights candidate frame the lower right corner coordinate；Or (x_top, y_top) is the upper right corner of traffic lights candidate frame Coordinate, (x_bot, y_bot) be traffic lights candidate frame the lower left corner coordinate.It can also be indicated with other multiple points, this Place is by way of example only.

For example: in conjunction with Fig. 3, input the information of the traffic lights candidate frame of present frame traffic lights image are as follows: traffic lights is waited The number for selecting frame is 5, and the index value ID of frame 1 is 1, and the position coordinates of frame 1 are (x_top1, y_top1, x_bot1, y_bot1), the index of frame 2 Value ID is 2, and the position coordinates of frame 2 are (x_top2, y_top2, x_bot2, y_bot2), the index value ID of frame 3 is 3, and the position coordinates of frame 3 are (x_top3, y_top3, x_bot3, y_bot3), and so on, the information of input frame 4 and frame 5.

Step 2: clustering processing being carried out to the traffic lights candidate frame of input, the condition of cluster is to calculate any two traffic lights Two traffic lights candidate frames that distance is no more than the first preset threshold are divided into same first square by the distance between candidate frame In shape frame.A bit of same position i.e. on two traffic lights candidate frames of selection, the distance between two o'clock is that two traffic lights are waited Select the distance between frame.Such as: two are calculated according to the coordinate (x_top, y_top) in any two traffic lights candidate frame upper left corner The distance between traffic lights candidate frame.

Specific clustering algorithm is as follows:

If the distance between two traffic lights candidate frames d is less than the first preset threshold, by two traffic lights candidate frames It is divided into same first rectangle frame, as shown in figure 4, for frame 1, frame 2, frame 3, frame 4 and the frame 5 in present frame traffic lights image The schematic diagram for carrying out the first rectangle frame obtained after clustering processing is determined according to clustering algorithm, any two in frame 1, frame 2 and frame 3 The distance between a frame is both less than the first pre-determined distance, the distance between frame 4 and frame 5 again smaller than the first pre-determined distance, but frame 1, Frame 2 and the distance between frame 4 and frame 5 are both greater than the first pre-determined distance, and a traffic lights candidate frame can be only present in one the In one rectangle frame, therefore frame 1, frame 2 and frame 3 are divided into the first rectangle frame 1, therefore frame 4 and frame 5 are divided into the first rectangle frame In 2.Wherein, the first rectangle frame 1 is identical with the specification of the first rectangle frame 2, is all a length of H, and width is the rectangle frame of W, can be in this way Understand that the first rectangle frame 1 can be by being rotationally and/or translationally overlapped with the first rectangle frame 2.

Wherein, the position of the first rectangle frame according to comprising the position of the second rectangle frame determine, such as: in conjunction with Fig. 4, first It include frame 1, frame 2 and frame 3, the top left corner apex coordinate weight of the top left corner apex coordinate and frame 1 of the first rectangle frame 1 in rectangle frame 1 It closes.

Second aspect: color state and the position of the traffic lights in present frame traffic lights image are detected by neural network model In the position of the first rectangle frame；

The first rectangle frame for including in present frame traffic lights image is determined through the above steps, it will be corresponding in current frame image The location information of each first rectangle frame be input in neural network model, successively detect each first by the neural network model The color state of traffic lights in rectangle frame is red light, green light or amber light, and detects the traffic lights and be located in the first rectangle frame More accurate position.

Wherein, it is illustrated in figure 5, the color state of each traffic lights and position in the first rectangle is determined by neural network model The main flow of confidence breath, comprising the following steps:

Step 500: the location information of the first rectangle frame is input in neural network model；

Since the first rectangle frame being input in neural network model is preset fixed specification, the first square is being inputted When the location information of shape, the position of the first rectangle frame can be indicated by the coordinate on the left vertex of the first rectangle frame, such as: knot Fig. 2 is closed, the coordinate on the left vertex of the first rectangle frame 1 and the coordinate on the left vertex of the first rectangle frame 2 are inputted into neural network model, mind Successively the first rectangle frame 1 and the first rectangle frame 2 are detected through network model, determine the traffic when acquiring traffic lights image The actual color state of lamp and the position in the first rectangle frame, the detection for color state, for example, in traffic lights image The color that the traffic lights is shown is amber light, and actual color state is red light.For position detection, for example, determining traffic lights Minimum circumscribed rectangle frame, export coordinate of the central point of the minimum circumscribed rectangle frame in the first rectangle frame and the minimum it is external The length and width value of rectangle frame.

Step 501: data conversion being carried out to the corresponding image of the first rectangle frame, the image data after conversion is inputted into nerve In network model；

The corresponding YUV image of first rectangle frame is converted into RGB image, by the RGB image of the first rectangle frame after conversion It is input in neural network model, wherein can be converted the yuv data of the corresponding image of the first rectangle frame by following equation At RGB image:

R=Y+1.402*V

G=Y-0.344*U-0.714*V

B=Y+1.772*U

Step 502: neural network model detects in the first rectangle frame according to the location information and RGB data of the first rectangle frame Traffic lights color state and traffic lights be located at the location information in the first rectangle frame；

Neural network model in the embodiment of the present invention is made of multiple convolutional layers, pond layer, up-sampling layer and cascading layers, It is illustrated in figure 6 the knot of the neural network model of the color state provided in an embodiment of the present invention for detecting traffic lights and position It is as follows to predominantly detect process for structure:

1) firstly, carrying out first layer convolution operation to the rgb image data of input, filter size is r × r, and r is 1~7 Between number, export c1 channel characteristics figure, c1 is the number between 8~1024, and level 1 volume product exports result are as follows:

Wherein,Indicate convolution operation, I_iIndicate i-th of the channel R, G, B, w_1,i,jIndicate j-th channel of level 1 volume product the The weight of i filter, b_1,jIndicate j-th of the biasing of level 1 volume product, F_1,jIt is j-th of channel output result of level 1 volume product；

2) activation processing is carried out to the result of first layer convolution operation, activation primitive is as follows:

F_1,jResult after activation are as follows:

Wherein, α indicates gain coefficient, for controlling F_1,jPart less than 0, α is between 0~1；Wherein, F1, j are the 1st The output result in j-th of channel of layer convolution.

3) by the activation of the result of first layer convolution operation, treated that data carry out pond processing；

4) processing of level 2 volume product, the activation processing of level 2 volume product result are successively executed and to level 2 volume product after activation The result of operation carries out pond processing, and so on, n-th process of convolution of going directly is completed.

5) up-sampling treatment is carried out to the result of n-th process of convolution；

Assuming that characteristic pattern size is W after n-th layer process of convolution_n×H_n×C_n, then output size after up-sampling are as follows: W_n×N× H_n×N×C_n, wherein N is up-sampling multiple；

Wherein, sampling processing is considered as the phase inverse processing in pond, is in fact exactly image amplification, such as the input of up-sampling It is the image of 6x6 size, up-sampling multiple is 2, then up-sampling output is the image of 12x12.

6) cascade operation is carried out after up-sampling treatment；

Cascade is exactly by the result and up-sampling result progress channel splicing after the convolution of t layers of front, it is assumed that t layers of convolution Result afterwards is W_t×H_t×C_t, up-sampling output result is W_t×H_t×C_up1, then result after cascading are as follows: W_t×H_t×(C_t+ C_up1), circulation executes upper acquisition operation and the cascade operation of setting number later, finally executes process of convolution.

7) according to the output of the last layer process of convolution as a result, calculating confidence level highest using non-maxima suppression algorithm Traffic lights position coordinates and score, the color state of traffic lights is obtained according to score, judges the color state of Current traffic lamp It is red light, amber light or green light, while the minimum circumscribed rectangle frame for obtaining each traffic lights is located at the position in the first rectangle frame Coordinate exports the color state of each traffic lights and the position coordinates and length of corresponding minimum circumscribed rectangle frame in the first rectangle frame Width values.

It can detecte into the first rectangle frame each traffic lights corresponding color state by above-mentioned processing and be located at the first rectangle In compared to the more accurate band of position in traffic lights candidate region, in order to which whether the position for detecting neural network model output accurate Also need further to detect, the band of position of traffic lights neural network model exported below by the content of Part III into The detailed process of row detection is introduced.

The third aspect: position judgement is carried out to the traffic lights that the color state detected is red light；

As shown in fig. 7, traffic lights 1 is located in the first rectangle frame 1 in the first rectangle frame 1 detected for neural network model Location information, wherein (x_mid, y_mid) be traffic lights 1 minimum external matrix frame central point relative to the first rectangle frame 1 The long a of coordinate and the minimum circumscribed rectangle frame and width b.

If neural network model detects the color state of traffic lights 1 for red, according to the friendship of neural network model output The above-mentioned location information of logical lamp 1, can determine the band of position that traffic lights 1 is located in the first rectangle frame 1, i.e., as shown in Figure 7 The minimum circumscribed rectangle frame of traffic lights 1 is to be the traffic lights 1 that is detected according to neural network model shared by the first rectangle frame 1 The band of position (i.e. first position region).

It is located at position when color state in preceding N frame traffic lights image is red light according to the traffic lights 1 and determines the traffic lights 1 The corresponding band of position (i.e. the second position region of the traffic lights 1), by the first position region of the traffic lights 1 and the second position Region is compared, to judge whether the first position region of traffic lights 1 of neural network model output is accurate.

Wherein it is determined that being exemplified below there are many modes of the second position of traffic lights 1:

Method of determination one: the second position region of real-time update traffic lights；

It is illustrated in figure 8 the mode of the second position of real-time update traffic lights, comprising the following steps:

Step 800: determining color state and the position of the traffic lights 1 in corresponding first rectangle frame of m frame traffic lights image It sets；

Step 801: determine the traffic lights 1 in corresponding first rectangle frame of m+1 frame traffic lights image color state and Position；

Step 802: according to the location information and m+1 frame traffic lights image of the corresponding traffic lights 1 of m frame traffic lights image The location information of corresponding traffic lights 1 is weighted fusion, determines the second position region i of traffic lights 1；

Step 803: determine the traffic lights 1 in corresponding first rectangle frame of m+2 frame traffic lights image color state and Position；

Step 804: according to the location information of the corresponding traffic lights 1 of m+2 frame traffic lights image and second position region i into Row Weighted Fusion obtains the corresponding second position region i+1 of traffic lights 1.

For example: assuming that being detected in the first frame traffic lights image of Traffic Camera acquisition by neural network model The color state and location information of each traffic lights in first rectangle frame 1, if detecting, the color state of the traffic lights 1 is red light, Then the location information 1 of the traffic lights 1 is saved；

The first rectangle frame 1 in the second frame traffic lights image of same Traffic Camera acquisition is detected by neural network model, If the color state of the traffic lights 1 is also red in the second frame traffic lights image, it is somebody's turn to do what neural network model detected The location information 2 of traffic lights 1 is saved；

The band of position 1 that location information 1 determines is compared with the band of position 2 that location information 2 determines, calculating position Region 1 is at a distance from the band of position 2, if the distance is less than the second preset threshold, by the band of position 1 and the band of position 2 into Row Weighted Fusion obtains the 1 corresponding second position region (hereinafter referred to as reference position region 1) of traffic lights.In Fig. 9 (a) It is shown, the location information 1 of the corresponding traffic lights 1 of first frame traffic lights image is determined for neural network model；(b) institute in Fig. 9 Show, the location information 2 of the corresponding traffic lights 1 of the second frame traffic lights image is determined for neural network model；(c) is to pass through in Fig. 9 The second location information for the traffic lights 1 that location information 1 and location information 2 determine.

It should be noted that if the band of position 1 at a distance from the band of position 2 more than the second preset threshold, then without The update of two bands of position is handled.

If the Traffic Camera acquires third frame traffic lights image at this time, the traffic of third frame is detected by neural network model First rectangle frame 1 in lamp image will be refreshing if the color state of the traffic lights 1 is also red in third frame traffic lights image The location information 3 of the traffic lights 1 detected through network model is saved；

The band of position 3 that location information 3 determines is compared with reference position region 1, calculating position region 3 and reference The distance of the band of position 1 is added the band of position 3 and reference position region 1 if the distance is less than the second preset threshold Power fusion, obtains the corresponding reference position region 2 of traffic lights 1, and update the corresponding second position region of traffic lights 1.

And so on, when the color state for often detecting a satisfactory traffic lights 1 is the traffic lights image of red light, The location information of the traffic lights 1 determined according to corresponding first rectangle frame of the frame traffic lights image with traffic lights 1 corresponding the Two bands of position are weighted fusion, and update the corresponding second position region of traffic lights 1 out.

Method of determination two: the position for the traffic lights 1 that N frame traffic lights image determines is weighted fusion；

When determining traffic lights 1 color state being red light in preceding N frame traffic lights image, the corresponding band of position, and by preceding N The corresponding band of position of traffic lights 1 is weighted fusion in frame, obtains the corresponding second position region of traffic lights 1.

Such as: assuming that N is 3, present frame traffic lights image is the 6th frame, and determines that traffic lights 1 exists by neural network model Color state in 5th frame, the 4th frame and the 3rd frame traffic lights image is all red light, the corresponding position in the 5th frame traffic lights image Setting region is band of position a, and the corresponding band of position is band of position b in the 4th frame traffic lights image, in the 3rd frame traffic lights The corresponding band of position is band of position c in image, then is weighted meter to band of position a, band of position b, band of position c It calculates, determines the corresponding second position region of traffic lights 1.

If the color state of the traffic lights 1 is that the frame number of red light is less than N in the traffic lights image before current frame image, The process that the location information of the traffic lights of neural network model output is compared to judgement with second position region is not executed, directly The band of position for determining the location information of neural network model output is connect as target position region.

If the color state of the traffic lights 1 is that the frame number of red light is not less than N in the traffic lights image before current frame image, The first position region and preceding N frame traffic lights image that then the location information of the traffic lights of neural network model output is determined are determining The second position region of the traffic lights be compared, judge first position region at a distance from the region of the second position whether be more than Second preset threshold, if being no more than the second preset threshold, using first position region as including in present frame traffic lights image The target position region of pixel to be processed；Otherwise, being by second position region includes picture to be processed in present frame traffic lights image The target position region of element.

Wherein it is determined that there are many modes of the distance in first position region and second position region, for example, selection first The same datum mark on region and second position region is set, calculates the distance between two datum marks, such as according to first position The minimum of the central point of the minimum circumscribed rectangle frame of the corresponding traffic lights in region traffic lights corresponding with second position region is external The central point of rectangle frame calculates distance of the distance between two central points as the first rectangular area and the second rectangular area.

After target position region determines, it is thus necessary to determine that go out pixel to be processed, i.e. target position region in the region of target position The mark such as interior round traffic lights, number, arrow, below only to have a kind of traffic lamp group that color state traffic lights forms to be Example, determines that the mode of pixel to be processed carries out to the brightness according to each pixel in the region of target position that the embodiment of the present invention provides Introduce explanation:

It as shown in Figure 10, wherein include corresponding second rectangle frame of traffic lamp group of three traffic lights according to Figure 10 (a) The schematic diagram of the first determining rectangle frame, wherein the traffic lamp group is from respectively straight trip red light, straight trip amber light and straight trip are green from left to right Lamp；Black region is non-bright area, and white area is that the straight trip red arrow identifies corresponding highlight regions.10 (b) be that will scheme First rectangle frame shown in 10 (a) is input to the traffic lamp group that neural network model obtains corresponding in the first rectangle frame One band of position, if after being compared according to the second position corresponding with traffic lights region, determining first position region for packet Target position region containing pixel to be processed determines that brightness is more than the according to the brightness of each pixel in the target position region The pixel of three preset thresholds is pixel to be processed, and the white area as shown in Figure 10 (b) is the corresponding region of straight trip red arrow, The region of red pixel composition i.e. to be coated.

Whether the brightness for judging each pixel in the region of target position is more than third predetermined threshold value, if being more than, it is determined that brightness It is pixel to be processed more than the pixel of third predetermined threshold value.Further, the gray value of pixel to be processed can also be set to 255, The gray value that brightness is less than the pixel of third predetermined threshold value is set to 0, generates the corresponding binary map in target position region, is led to Morphological scale-space is crossed, the noise spot in binary map is removed, obtains more accurate pixel to be processed.Wherein, Morphological scale-space There are many modes, such as the expansive working in image procossing.

Further, the embodiment of the invention also provides a kind of modes of noise spot in removal traffic lamp group, as shown in figure 11, The first rectangle frame 1 that collected certain the traffic lights image of Traffic Camera determines, straight trip red light arrow attachment has in the traffic lights group The barriers such as similar branch block, and the barrier is in reflective state in shooting at night, so that passing through the frame traffic lights The pixel to be processed that the corresponding target position region of image determines is that straight trip red light haircut identifies corresponding pixel and branch is corresponding Pixel, lead to determining pixel to be processed inaccuracy, precision is low, and the pixel to be processed determined to this part carries out applying red operation Afterwards, can the barriers such as branch also can Tu Hong, cause image procossing to be distorted.The embodiment of the invention provides a kind of removal traffic lights Corresponding first rectangle frame 1 of present frame traffic lights image is input in neural network model, if detecting this by the mode of interference When traffic lamp group is green light, i.e., as shown in (a) in Figure 12, the corresponding pixel of traffic lamp group when according to straight trip green light is built Background model is found and is updated, the formula of background model is updated are as follows:

Background (t)=(1-w) * background (t-1)+w*pixel (t)

Wherein background (t-1) is former frame background model, and background (t) is present frame background model, Pixel (t) is current pixel point value, and w is learning rate, and w is bigger, and the speed that background model updates is faster.

As shown in (b) in Figure 12, when neural network model detects that the traffic lamp group in the first rectangle frame 1 is that red light is bright When, by (b) of Figure 12 the corresponding minimum circumscribed rectangle frame of straight trip red light with according to the background mould that updates when straight trip green light The corresponding minimum circumscribed rectangle frame of red light of keeping straight on accordingly in type, which is cut out, to be come, as shown in figure 13, will be red in the region of target position Each pixel is compared in the minimum circumscribed rectangle frame of each pixel red light corresponding with background model in the minimum circumscribed rectangle frame of lamp It is right, if the luminance difference of pixel is more than third predetermined threshold value, it is determined that the pixel is pixel to be processed, i.e. white in (c) in Figure 13 The corresponding pixel in color region, the interference of night obstruction object is removed with this.

After the completion of pixel to be processed determines, color correction is carried out to the pixel to be processed, is carried out below by fourth aspect It is discussed in detail.

Fourth aspect: color correction is carried out to pixel to be processed.

The corresponding average brightness of pixel region to be processed is calculated, y_avg is denoted as, average brightness y_ is determined by following equation The corresponding color correction grade of avg；

Wherein, x₁、x₂、y₁、y₂、D₁、D₂、D₃It is constant；

Later, traverse pixel in the red light region detected, when the pixel corresponding R, G be greater than threshold value thr and Thg, it is determined that the pixel to be processed carries out applying red processing, and the RGB of pixel to be processed is converted to HSV, adjusts the pixel The value of H, S of point, wherein specific adjustment mode is as follows:

1) H value is adjusted；

If average brightness degree=0 keeps the H of pixel constant；

If average brightness degree ≠ 0, the H of pixel is adjusted to default H value.

2) S value is adjusted, it is assumed that m1=a, m2=1/b；

If average brightness degree=0 keeps the S of pixel constant；

If the mode for adjusting S is as follows when average brightness degree > 0:

If S >=degree, adjusting S is a*S, and returns to execution judges the step of whether S is more than or equal to degree, until The value of S is not less than degree or until reaching the setting number for adjusting S；

If S < degree, adjusting S is S/b.

After the adjustment of H, S value for completing pixel to be processed, HSV is converted into RGB.

It is above-mentioned to be red light to one of color state in corresponding first rectangle frame 1 of present frame traffic lights image The image processing flow of traffic lights, for the process flow for the traffic lights that other color states in the first rectangle frame 1 are red light It may refer to above-mentioned process, details are not described herein again, correspondingly, other first rectangle frames corresponding for present frame traffic lights image Processing may refer to the processing for above-mentioned first rectangle frame 1, when to all first rectangles in present frame traffic lights image After the completion of frame processing, terminate to present frame traffic lights image procossing.

Based on identical inventive concept, a kind of equipment of traffic lights image procossing is additionally provided in the embodiment of the present invention, such as Shown in Figure 14, which includes: at least one processing unit 1400 and at least one storage unit 1401, wherein the storage Unit 1401 is stored with program code, when said program code is executed by the processing unit 1400, so that the processing is single Member 1400 executes following process:

Optionally, first rectangle frame includes at least one second rectangle frame, and same first rectangle frame includes The distance between the second rectangle frame of any two less than the first preset threshold, second rectangle frame includes described at least one Traffic lights in traffic lights image.

Optionally, the processing unit 1400 determines first rectangle frame in the following manner:

Optionally, the processing unit 1400 is specifically used for:

Optionally, the processing unit 1400 is also used to:

Optionally, the processing unit 1400 adjusts each pixel or described that the target position includes in the following manner The HS value of target pixel points to be processed:

Determine the average brightness value of the pixel to be processed；

Based on identical design, also go out setting to another traffic lights image procossing as shown in figure 15 for the embodiment of the present invention Standby structural schematic diagram, the equipment include:

First determining module 1500: including first of the traffic lights in present frame traffic lights image for being directed to any one Rectangle frame determines each traffic lights for including in first rectangle frame in first rectangle frame by neural network model First position region and the color state for acquiring the traffic lights when traffic lights image；

Second determining module 1501: for determining that the color state of the neural network model output is the traffic lights of red light Corresponding second position region when color state is red light in preceding N frame traffic lights image；

Comparison module 1502: for by the first position region of the same traffic lights and the second position region into Row compares；If the first position region of the traffic lights is no more than the second default threshold at a distance from the second position region Value, it is determined that the first position region is the target position region in present frame traffic lights image comprising pixel to be processed；If The first position region of the traffic lights is at a distance from the second position region more than the second preset threshold, it is determined that institute Stating second position region is the target position region in present frame traffic lights image comprising pixel to be processed.

Optionally, first determining module 1500 determines first rectangle frame in the following manner:

Optionally, first determining module 1500 is specifically used for:

Optionally, second determining module 1501 is specifically used for:

Optionally, the comparison module 1503 is also used to:

After determining the first position region for comprising the target position region of pixel to be processed, the target is determined The brightness for the pixel for including in the band of position is more than that the pixel of third predetermined threshold value is pixel to be processed；

Optionally, the comparison module 1503 adjusts each pixel or described that the target position includes in the following manner The HS value of target pixel points to be processed:

Determine the average brightness value of the pixel to be processed；

Based on the same technical idea, the embodiment of the present application also provides a kind of computer readable storage mediums.The meter Calculation machine readable storage medium storing program for executing is stored with computer executable instructions, and the computer executable instructions are for holding the computer Process in the method for traffic lights image procossing in row previous embodiment.

Above by reference to showing according to the method, apparatus (system) of the embodiment of the present application and/or the frame of computer program product Figure and/or flow chart describe the application.It should be understood that can realize that block diagram and or flow chart is shown by computer program instructions The combination of the block of a block and block diagram and or flow chart diagram for figure.These computer program instructions can be supplied to logical With computer, the processor of special purpose computer and/or other programmable data processing units, to generate machine, so that via meter The instruction that calculation machine processor and/or other programmable data processing units execute creates for realizing block diagram and or flow chart block In specified function action method.

Correspondingly, the application can also be implemented with hardware and/or software (including firmware, resident software, microcode etc.).More Further, the application can take computer usable or the shape of the computer program product on computer readable storage medium Formula has the computer realized in the medium usable or computer readable program code, to be made by instruction execution system It is used with or in conjunction with instruction execution system.In the present context, computer can be used or computer-readable medium can be with It is arbitrary medium, may include, stores, communicates, transmits or transmit program, is made by instruction execution system, device or equipment With, or instruction execution system, device or equipment is combined to use.

Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art Mind and range.In this way, if these modifications and changes of the present invention belongs to the range of the claims in the present invention and its equivalent technologies Within, then the present invention is also intended to include these modifications and variations.

Claims

1. a kind of method of traffic lights image procossing, which is characterized in that this method comprises:

It include the first rectangle frame of the traffic lights in present frame traffic lights image for any one, it is true by neural network model First position region and the acquisition friendship of each traffic lights for including in fixed first rectangle frame in first rectangle frame The color state of traffic lights when logical lamp image；

Determine neural network model output color state be red light traffic lights in preceding N frame traffic lights image color shape State corresponding second position region when being red light；

If the first position region of the traffic lights is no more than the second preset threshold at a distance from the second position region, Then determine that the first position region is the target position region in present frame traffic lights image comprising pixel to be processed；

If the first position region of the traffic lights at a distance from the second position region more than the second preset threshold, Determine that the second position region is the target position region in present frame traffic lights image comprising pixel to be processed.

2. the method as described in claim 1, which is characterized in that first rectangle frame includes at least one second rectangle frame, The distance between second rectangle frame of any two that same first rectangle frame includes is less than the first preset threshold, and described Two rectangle frames include the traffic lights at least one described traffic lights image.

3. method according to claim 2, which is characterized in that determine first rectangle frame in the following manner:

According in the traffic lights image on the second rectangle frame of any two same position a bit, determine two the second rectangle frames The distance between；

The second rectangle frame where distance to be no more than to the point of the first preset threshold is divided into same first rectangle frame；Its In, the second rectangle frame is located in first rectangle frame；One of them second square that first rectangle frame and the first rectangle frame include The vertex of the same orientation of shape frame is overlapped.

4. the method as described in claim 1, which is characterized in that described to determine first rectangle frame by neural network model First position region of each traffic lights for inside including in first rectangle frame, comprising:

According to the central point in the corresponding first position region of each traffic lights of neural network model output in first square The dimension information of coordinate and the first position region in shape frame determines the traffic lights in first rectangle frame First position region.

5. the method as described in claim 1, which is characterized in that the color state of the determination neural network model output For traffic lights corresponding second position region when color state is red light in preceding N frame traffic lights image of red light, comprising:

The color state to neural network model output is that the traffic lights of red light is located at face in preceding N frame traffic lights image Color state is weighted in corresponding first position region when being red light, obtains the corresponding second position area of the traffic lights Domain.

6. the method as described in claim 1, which is characterized in that the determination first position region is to include picture to be processed After the target position region of element, further includes:

Determine the pixel for including in the target position region brightness be more than third predetermined threshold value pixel be pixel to be processed；

The HS value of the pixel to be processed is adjusted, so that color adjusted is shown as the acquisition that the neural network model determines The actual color state of the traffic lights when traffic lights image.

7. method as claimed in claim 6, which is characterized in that adjust each picture that the target position includes in the following manner The HS value of the plain or described target pixel points to be processed:

Determine the average brightness value of the pixel to be processed；

If the color correction parameters are not 0, the H value for adjusting the pixel to be processed is default H value；And judge described wait locate Whether the S value of reason pixel is less than the color correction parameters；

If it is determined that the S value of the pixel to be processed is less than the color correction parameters, then the S value of the pixel to be processed is adjusted It is m1 times of current S value, and returns to execution and judge whether the S value of the pixel to be processed is less than the step of the color correction parameters Suddenly, the S value of the pixel to be processed is adjusted not less than color correction parameters or reach until the S value of the pixel to be processed Preset times；

If it is determined that the S value of the pixel to be processed is not less than the color correction parameters, then by the S value tune of the pixel to be processed Whole m2 times for current S value.

8. a kind of equipment of traffic lights image procossing, which is characterized in that the equipment includes: at least one processing unit and at least one A storage unit, wherein the storage unit is stored with program code, when one or more computers of memory storage When program is executed by the processor, so that the equipment executes following process:

9. equipment as claimed in claim 8, which is characterized in that wherein, first rectangle frame includes at least one second square Shape frame, the distance between second rectangle frame of any two that same first rectangle frame includes less than the first preset threshold, Second rectangle frame includes the traffic lights at least one described traffic lights image.

10. equipment as claimed in claim 8, which is characterized in that the processor determines first square in the following manner Shape frame:

The second rectangle frame where distance to be no more than to the point of the first preset threshold is divided into same first rectangle frame；Its In, the second rectangle frame is located in first rectangle frame.

11. equipment as claimed in claim 8, which is characterized in that the processor is specifically used for:

12. equipment as claimed in claim 8, which is characterized in that the processor is specifically used for:

13. equipment as claimed in claim 8, which is characterized in that the processor is also used to:

14. equipment as claimed in claim 13, which is characterized in that the processor adjusts the target position in the following manner Set the HS value of each pixel for including or the target pixel points to be processed:

Determine the average brightness value of the pixel to be processed；