CN107730904A

CN107730904A - Multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks

Info

Publication number: CN107730904A
Application number: CN201710440989.1A
Authority: CN
Inventors: 汤平; 汤一平; 王辉; 温晓岳; 钱小鸿
Original assignee: Enjoyor Co Ltd
Current assignee: Enjoyor Co Ltd
Priority date: 2017-06-13
Filing date: 2017-06-13
Publication date: 2018-02-23

Abstract

A kind of multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks, including video camera, traffic Cloud Server and road vehicle reverse driving detecting system on urban road；Traffic Cloud Server is used to receive the road video data obtained from video camera, and be submitted to road vehicle reverse driving detecting system and detected and identified, road and direction of traffic customized module are included in road vehicle reverse driving detecting system, based on Faster R CNN vehicles and yellow line detection module, optical flow method direction of vehicle movement detection module, vehicle driving in reverse determination module, vehicle type recognition module, License Plate and identification module and report on infringement of regulations generation module.The present invention can effectively improve the accuracy of detection and robustness of reverse driving illegal activities, while also improve the level of vehicles peccancy identification capability and automatic business processing traffic offence event.

Description

Multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks

Technical field

The present invention relates to artificial intelligence, convolutional neural networks and computer vision answering in vehicle driving in reverse context of detection With belonging to intelligent transportation field.

Background technology

Bicycle road plays important role in urban road is current, improves the current level of urban road, effectively Alleviate urban traffic blocking.Increasingly strengthen however as one-lane effect, adjoint potential safety hazard is increasingly by people Attention, and the violation event that drives in the wrong direction is the major reason for causing the current potential safety hazard of bicycle road, how effectively to find to drive in the wrong direction Phenomenon, and prevent in time, so as to prevent the personnel caused by driving in the wrong direction and property loss, always as in intelligent transportation field A hot issue, have huge commercial value and economic benefit.

Reverse driving, refer to vehicle illegal drive into to track, it is illegal drive into one-way road, in single file passage particularly people Retrograde behavior in stream aggregation area, easily causes traffic congestion, or even cause the accident.Current common detection means is according to car Road attribute, i.e., it is up or descending, the information such as wheelpath come judge whether it is illegal drive into to track, illegal drive into list The behaviors such as line.Therefore, the detection of driving in the wrong direction in real time carried out to the mobile target such as vehicle in monitoring scene, sends and alarms and trigger Other relevant actions, it is the main task detected of driving in the wrong direction.

The retrograde image detecting method of currently used vehicle has：Retrograde detection method based on light stream can detect object Motion, it is but sensitive to noise jamming, easily fail to report and report by mistake；Retrograde detection method based on target following can be relatively more steady Fixed detects to drive in the wrong direction, but needs to detect each target, and speed is slower, it is desirable to which tracking is continued for, and resource occupation is tight Weight, can not be applicable in the scene of multiple target；Retrograde detection method based on light stream and cluster can significantly reduce the complexity of computing Degree, but only determine to drive in the wrong direction by clustering optical flow field characteristic point, the accuracy rate of detection is again insufficient.

Chinese invention patent application number discloses a kind of car lane car based on image procossing for 201610319456.3 Retrograde recognition methods, this method improve center yellow line using geometry filtering and spatial clustering method filtering line segment noise Discrimination；Pre-tracking and prediction are carried out to center yellow line using Kalman filter algorithm simultaneously so as to draw optimal center yellow line, Influence caused by the center yellow line identification mistake for significantly reducing indivedual frames or recognition methods of unsuccessfully being driven in the wrong direction to vehicle

Chinese invention patent application number is 201510089718.7 detection method and the detections for disclosing that a kind of vehicle drives in the wrong direction System, the device mainly determine from road driving image vehicle to be detected in the road by colour recognition and Straight Line Identification Position, the travel direction of estimating of vehicle is determined in conjunction with the static nature thing for characterizing direction, and detected and moved by optical flow method The direction of motion and speed of dynamic characteristic body, finally weighting determine whether vehicle drives in the wrong direction.

The vision detection technology in deep learning epoch before above-mentioned vision detection technology belongs to, there is accuracy of detection and detection The problem of robustness is not high.

Recent years, deep learning in the technology of computer vision field obtained rapid development, and deep learning can utilize Substantial amounts of training sample and hidden layer successively in depth learn the abstracted information of image, more comprehensively directly obtain characteristics of image. Digital picture is described with matrix, and convolutional neural networks describe the whole of image preferably from local message block Body structure, therefore solve problem using convolutional neural networks mostly in computer vision field, deep learning method.Unroll Improve accuracy of detection and detection time, depth convolutional neural networks technology is from R-CNN, Faster R-CNN to Fasterer R- CNN.Be embodied in further precision improvement, acceleration, end-to-end and more practical, almost cover from be categorized into detection, point Cut, position every field.Depth learning technology applies to vehicle driving in reverse vision-based detection and will be one to have very much practical application The research field of value.

When the vision system of the mankind is perceiving moving target, moving target can be formed on the imaging plane of vision system A kind of image stream of even variation, referred to as light stream.Light stream expresses image pixel and changed with time speed degree, is one The apparent motion of gradation of image pattern in image sequence, it is the pixel being observed on the surface motion of space motion object Instantaneous velocity field.The advantages of optical flow method, is the provision of the speed of related movement of moving target, exercise attitudes position and surface The abundant informations such as texture structure, and can be in the case where not knowing any information of scene, or even under complex scene, can also examine Measure moving target.Therefore, after road vehicle is detected, moving vehicle traffic direction can be identified with optical flow method.

The content of the invention

In order to overcome the shortcomings of that the accuracy of detection of existing vehicle driving in reverse detection mode is relatively low, detection robustness is not high, A kind of accuracy of detection of present invention offer is higher, the higher reverse row of multitask vehicle based on depth convolutional neural networks of robustness Sail vision detection system.

The technical solution adopted for the present invention to solve the technical problems is：

A kind of multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks, including installed in city Video camera, traffic Cloud Server and road vehicle reverse driving detecting system on city's road；

Described video camera is used to obtain the video data on each road in city, configures in the top of road, passes through network Vedio data on road is transferred to described traffic Cloud Server；

Described traffic Cloud Server is used to receive the road video data obtained from described video camera, and is passed Give described road vehicle reverse driving detecting system and carry out peccancy detection and to vehicles peccancy identification, finally will detection As a result merging scene candid photograph image automatically generates peccancy detection report and is stored in Cloud Server, so as to reverse driving vehicle Car owner is alerted and punished, so as to prevent the generation of reverse behavior；

Described road vehicle reverse driving detecting system includes road and direction of traffic customized module, based on FasterR- CNN vehicles and yellow line detection module, optical flow method direction of vehicle movement detection module, vehicle driving in reverse determination module, vehicle class Type identification module, License Plate and identification module and report on infringement of regulations generation module；

Described road and direction of traffic customized module are used for the track for customizing the road in camera field of view, in video figure Virtual track is drawn according to the lane on real road and direction of traffic as in, virtual lane markings go out the traveling side of vehicle To；

Described Faster R-CNN are the depth convolutional neural networks of a multitask, are realized first in the network Detection positioning to road vehicle and yellow line, then the type of vehicles peccancy is identified, then to vehicles peccancy Car plate is positioned and identified that detection fixation and recognition shares the convolutional layer of same depth convolutional neural networks.

Further, it is described to be used to detect in video image based on Faster R-CNN vehicles and yellow line detection module Yellow line on all vehicles and road, specific practice are the motor vehicles gone out using depth convolutional neural networks Fast Segmentation on road And yellow line and provide these vehicles and yellow line spatial positional information shared on road；

Yellow line segmentation and positioning on motor vehicle and road used herein are made up of two models, and a model is generation RoI selective search network；Another model is the yellow line target detection network on Faster R-CNN motor vehicles and road, Detection unit structure chart is as shown in Figure 1.

Selective search network, i.e. RPN；RPN networks are built any scalogram picture as input, output rectangular target The set of frame is discussed, each frame includes 4 position coordinates variables and a score.For formation zone Suggestion box, at last Small network is slided in the convolution Feature Mapping of shared convolutional layer output, this network is connected to input convolution Feature Mapping entirely In n × n spatial window.Each sliding window is mapped on a low-dimensional vector, a sliding window of each Feature Mapping A corresponding numerical value.This vector exports the layer of the full connection at the same level to two.

In the position of each sliding window, while k suggestion areas is predicted, so position returns layer and has 4k output, The codes co-ordinates of i.e. k bounding box.Layer of classifying exports the score of 2k bounding box, i.e., is target/non-targeted to each Suggestion box Estimated probability, be the classification layer realized with the softmax layers of two classification, k can also be generated with logistic recurrence Point.K Suggestion box is parameterized by the corresponding k Suggestion box for being referred to as anchor.Each anchor is with current sliding window mouth center Centered on, and a kind of corresponding yardstick and length-width ratio, using 3 kinds of yardsticks and 3 kinds of length-width ratios, so just have in each sliding position K=9 anchor.For example, for the convolution Feature Mapping that size is w × h, then a total of w × h × k anchor.RPN nets Network structure chart is as shown in Figure 2.

In order to train RPN networks, a binary label is distributed to each anchor, is to mark the anchor with this It is not target.Then distribute positive label and give this two class anchor：(I) with some real target bounding box, i.e. Ground Truth, GT has the ratio between highest IoU, i.e. Interse-ction-over-Union, common factor union, overlapping anchor；(II) it is and any GT bounding boxs have the overlapping anchor of the IoU more than 0.7.Notice that a GT bounding box may give multiple anchor distribution positive mark Label.The IoU ratios that the negative label of distribution gives all GT bounding boxs are below 0.3 anchor.Anon-normal non-negative anchor is to instruction Practicing target does not have any effect, then abandons.

There are these to define, it then follows the multitask loss in Faster R-CNN, to minimize object function.To an image Loss function be defined as：

Here, i is anchor index, p_iIt is the prediction probability that anchor is the i-th target, if anchor is Just, GT labelsIt is exactly 1, if anchor is negative,It is exactly 0；t_iIt is a vector, represents 4 ginsengs of the bounding box of prediction Numberization coordinate,It is the coordinate vector of GT bounding boxs corresponding with positive anchor；λ is a balance weight, here λ=10, N_cls The normalized value for being cls items is mini-batch size, here N_cls=256, N_regThe normalized value for being reg items is anchor The quantity of position, N_reg=2,400；Classification Loss function L_clsTwo classifications, i.e. motor vehicles target and non power driven vehicle mesh Mark and the logarithm of yellow line target and road background lose：

For returning loss function L_reg, defined to minor function：

In formula, L_regTo return loss function, R is the loss function of robust, and smooth L are calculated with formula (4)₁；

In formula, smooth_L1For smooth L₁Loss function, x are variable；

Faster R-CNN network structures in input picture after depth convolutional neural networks as shown in figure 3, can obtain To characteristic pattern, corresponding RoIs can be then obtained according to characteristic pattern and RPN networks, finally then passes through RoI ponds layer.The layer is The only process in level spatial " pyramid " pond.Input is N number of Feature Mapping and R RoI.N number of Feature Mapping comes from most The latter convolutional layer, the size of each Feature Mapping is w × h × c.Each RoI is a tuple (n, r, c, h, w), wherein, N is the index of Feature Mapping, and n ∈ (0,1,2 ..., N-1), r, c are top left co-ordinates, and h, w are height and width respectively.Output then by The Feature Mapping that maximum pond obtains.The effect of this layer mainly has two, first, by the block pair in the RoI and characteristic pattern in artwork It should get up；It by characteristic pattern down-sampling is fixed size that another, which is, is then passed to full connection again.

Selective search network is shared with detecting the weights of network：Selective search network and Faster R-CNN are only Vertical training, differently to change their convolutional layer.Therefore need to allow to share convolution between two networks using a kind of The technology of layer, rather than learn two networks respectively.A kind of 4 practical step training algorithms are used in invention, pass through alternative optimization To learn shared feature.The first step, according to above-mentioned training RPN, the model initialization of network ImageNet pre-training, and hold It is used for region to end fine setting and suggests task.Second step, the Suggestion box generated using the RPN of the first step, is instructed by Faster R-CNN Practice an individually detection network, this detection network is equally by the model initialization of ImageNet pre-training, at this time Two networks are also without shared convolutional layer.3rd step, trained with detection netinit RPN, but fixed shared convolutional layer, and And only finely tune the exclusive layers of RPN, present two network share convolutional layers.4th step, keep shared convolutional layer to fix, finely tune Faster R-CNN fc, i.e., full articulamentum.So, two network share identical convolutional layers, a unified network is formed.

In view of object it is multiple dimensioned the problem of, use three kinds of simple chis for each characteristic point on characteristic pattern Degree, the area of bounding box is respectively 128 × 128,256 × 256,512 × 512 and three kind of length-width ratio, respectively 1:1、1:2、2: 1.Pass through this design, in this way it is no longer necessary to which Analysis On Multi-scale Features or multi-scale sliding window mouth predict big region, can reach section Save the effect of a large amount of run times.

By the processing of above-mentioned two network, detect yellow line on the motor vehicles and road in a frame video image simultaneously Their size and locus are confined, that is, obtained size and the locus of yellow line on vehicle and road, Its r_v,c_vIt is the top left co-ordinate of vehicle in the picture, h_v,w_vProjected size of the vehicle in the plane of delineation respectively, i.e., it is high and It is wide；Its r_y,c_yIt is the top left co-ordinate of yellow line in the picture on road, h_y,w_yIt is the yellow line on road respectively in the plane of delineation Projected size, i.e. height and width；Then need to judge the travel direction of these motor vehicles；

Because object of interest in the present invention is various motor vehicles and yellow line, i.e. object of interest, hereinafter referred to as RoI, will be various motor-driven in order to position and be partitioned into the various RoI on road, it is necessary in study and during training convolutional neural networks Yellow line and road background image on vehicle, road are put on corresponding label and are trained respectively；So in described base Motor vehicles and yellow line area-of-interest can be just partitioned into automatically in Faster R-CNN vehicles and the processing of yellow line detection module；

Further, described optical flow method direction of vehicle movement detection module is used to detect road vehicle traveling side To；When the vehicle in road scene correspond to two dimensional image plane move when, these vehicles two dimensional image plane projection just Motion is formd, the flowing that this motion is showed with plane of delineation luminance patterns is known as light stream.Optical flow method is to motion The important method that sequence image is analyzed, the movable information of Vehicle Object target in image is included in light stream；

The present invention uses a kind of sparse iterative method of Lucas-Kanade light streams based on pyramid model；Figure is first introduced below The pyramidal representation of picture, it is assumed that image I size is n_x×n_y.Define I⁰The 0th tomographic image is represented, the 0th tomographic image is rate respectively Highest image, i.e. original image, this tomographic image it is wide and a height ofWithThen we are with one Recursive mode is planted to describe pyramidal representation：We pass through I^L-1To calculate I^L(L=1,2 ...).I^L-1Represent pyramid L- 1 layer of image, I^LRepresent the image of pyramid L layers.Assuming that image I^L-1It is wide and a height ofWithSo image I^LCan be with It is expressed as

In order to simplify formula, we are by imageBoundary point value definition It is as follows,

The point that formula (5) defines must is fulfilled for conditionTherefore image I^LWidthAnd heightNeed to meet formula (6),

Image I pyramid model { I is built by formula (5) and (6)_LL=0 ..., L_m。L_mFor pyramid model Highly, L_mTypically take 2,3 or 4.For in general image L_mIt is just nonsensical more than 4.Using the image of 640 × 480 sizes as Example, the 1st, 2,3,4 tomographic image size of its pyramid model is respectively 320 × 240, and 160 × 120,80 × 60,40 × 30；

LK optical flow computation methods based on pyramid model, first the top k layer search characteristics in image pyramid model The match point of point, then kth -1 of the initial estimate in image pyramid model using the result of calculation of k layers as k-1 layers Layer search match point, goes round and begins again and iterates to the 0th layer of image pyramid model always, so as to which the light of this feature point be calculated Stream；

The detection target of optical flow method is：In front and rear two field pictures I and J, for image I some pixel u, in image Its match point v=u+d is found in J, or finds out its offset vector d, is calculated with formula (7)；

V=u+d=[u_x+d_x u_y+d_y]^T (7)

In formula, u is some pixel in image I, and v is pixel matched in image J, and d is between the two Offset vector；

First, image I and J pyramid model { I are established_LL=0 ..., L_m{ J_LL=0 ..., L_m；Then picture is calculated Vegetarian refreshments u positions in each pyramidal layers of image IL=0 ..., L_m；Then by a search window image J gold Word tower model highest tomographic image IL_mMiddle calculating uL_mMatch point vL_m, and calculate offset vector d^Lm。

Next we describe the optical flow method based on pyramid model with the mode of iteration；Assuming that pyramid mould is known The offset vector d of type L+1 layers^L+1.So by 2d^L+1As the initial value of L layers, with the match point for nearby searching for L layers vL；And then obtain the offset vector d of L layers^L；

By each layer of offset vector d of iterative calculation^LAfter (L=0 ..., Lm), the final light stream of the pixel is

In formula, d be a certain pixel light stream value, d^LFor a certain pixel L layers light stream value；

After the light stream vectors value of each feature pixel in obtaining image, according to described based on Faster R-CNN vehicles The motor vehicles on road and shared spatial positional information on road are detected with yellow line detection module, i.e., in two dimensional image The frame of each vehicle is obtained in plane, each frame has four data representations, the position r in the upper left corner_v,c_vAnd length and width h_v,w_v；Here the average value of all feature-point optical flow vectors of each inframe is calculated, is calculated with formula (9),

In formula,For the average value of the light stream vectors of certain vehicle inframe, d_iFor a certain feature pixel of certain vehicle inframe Light stream vectors value, n are the quantity of the feature pixel of certain vehicle inframe；

The average value of the light stream vectors of certain vehicle inframe is calculatedAfterwards, just should if the value is more than a certain threshold value T Vehicle frame is as moving vehicle, and the direction using the direction of the light stream vectors of the vehicle frame as moving vehicle；

In the system prerun stage, learn to obtain the direction of traffic in track automatically by self study mode；Specific practice is： First, by all vehicles detected based on Faster R-CNN vehicles and yellow line detection module in video image, Then, the light stream vectors for detecting these vehicles are calculated by described optical flow method direction of vehicle movement detection module, finally, These light stream vectors values are done into vector to add, the result that this vector adds to obtain can serve as the direction of traffic in track；System is pre- It is 100 that the time of operation phase will cross the number of vehicles of road depending on the number of vehicles of road is crossed, in the present invention, when reaching After this number, the system prerun stage just terminates, and system enters reverse driving monitor state；

Further, described vehicle driving in reverse determination module is used for the reverse row for the vehicle for identifying and judging road Behavior is sailed, there is two ways for reverse driving behavior：A kind of reverse driving behavior is in the road of defined driveway travel directions Reverse driving on road, for the judgement of the reverse driving behavior, pass through the row in the track that the system prerun stage obtains first Car direction, the light stream vectors of the vehicle of road were calculated according to described optical flow method direction of vehicle movement detection module, such as Angle between the direction of traffic in fruit track and the excessively light stream vectors of the vehicle of road is more than threshold value T_DIt is doubtful with regard to preliminary judgement Reverse driving behavior, T_D=150 °；

Another reverse driving behavior is to cross yellow line to enter inverted running track, for the judgement of this reverse behavior, It is whether to have yellow line on road to be detected first, also judges whether vehicle pushes down yellow line or excessively yellow if yellow line Line；When occurring due to such case, yellow line some parts are by occlusion on road, so needing yellow line part by algorithm Restore；Yellow line of the present invention on Hough transform fitting road, Hough transform are that P.V.Hough proposed in 1962 A kind of Shape Matching Technique, it the advantages of be strong antijamming capability, and there is high robust；The basic thought of Hough transform It is the antithesis row of point and line, the problem of former space is converted in its dual spaces and solved；But traditional Hough transform Due to needing to travel through point on all directions, thus it is computationally intensive, influence detection efficiency；Due to yellow line substantially with vehicle traveling side To parallel, therefore, the angle of yellow line traversal is limited in [- 5 ° ,+5 °] by the present invention；Tried to achieve in calculating on road after yellow line, then According to it is described based on Faster R-CNN vehicles and yellow line detection module detect Vehicle Object whether with yellow line have it is overlapping and Phenomenon is covered, if being doubtful reverse driving behavior with regard to preliminary judgement；

In order to facilitate collecting evidence and improving detection accuracy of identification, the present invention sets one using continuous candid photograph and identification technology Individual sign of flag, add 1 to this sign of flag when having detected reverse driving behavior, first time preliminary judgement is doubtful reverse During traveling behavior, Flag=1, piece image is captured；When still having detected reverse driving behavior for the second time, Flag=2, grab Clap the second width image；When third time has still detected reverse driving behavior, Flag=3, the 3rd width image is captured；Finally by this Three width images are stored in in the file of the name of time at that time；If do not detected reverse driving behavior, Flag= 0；

Described vehicle type recognition module is used to the type of vehicle of reverse driving be identified, to improve vehicles peccancy The level of identification capability and automatic business processing traffic offence event；Here multitask depth convolutional neural networks skill is used Art, it is described based on the Faster R-CNN networks of Faster R-CNN vehicles and yellow line detection module in add again in one Heart loss function and a softmax loss function, the joint-monitoring of center loss function and softmax loss functions is realized, Export to obtain the brand of vehicles peccancy, series, body color information by softmax；Described vehicle type recognition module is adopted With non real-time calculation, after described vehicle driving in reverse determination module is determined with reverse driving vehicle, start one Thread, the image that vehicles peccancy is captured with three width interior in the file of time name is read, type of vehicle is carried out to it respectively Identification；

Described License Plate and identification module are used to the car plate of the vehicle of reverse driving be identified, further to carry The level of high vehicles peccancy identification capability and automatic business processing traffic offence event；Here multitask depth convolution is used Nerual network technique, after described vehicle driving in reverse determination module is determined with reverse driving vehicle, start a thread, read The image that vehicles peccancy is captured with three width interior in the file of time name is taken, vehicles peccancy then is captured to three width respectively Car plate position on image is positioned, and obtains the image of car plate, then license plate image is identified, identify vehicles peccancy License plate number；

Described report on infringement of regulations generation module is used to automatically generate the report for punishing vehicles peccancy, according to described Brand, series, body color information, described License Plate and the knowledge for the vehicles peccancy that vehicle type recognition resume module obtains The license plate number for the vehicles peccancy that other module obtains, then according to these message reference vehicle managements vehicle registration database, Whether the vehicle identity information of matching identification and the vehicle identity information registered are consistent, and vehicles peccancy is automatically generated if consistent The report punished；If inconsistent, the information is pushed to administrative staff, further confirmed；Content in report Including：Image, vehicle identity information, place violating the regulations and the time that three width are captured when violating the regulations.

Beneficial effects of the present invention are mainly manifested in：Accuracy of detection is higher, robustness is higher.

Brief description of the drawings

Fig. 1 is Fast R-CNN structure charts；

Fig. 2 is selective search network；

Fig. 3 is Faster R-CNN structure charts；

Fig. 4 is Faster R-CNN multitask vehicle driving in reverse vision-based detection network structures；

Fig. 5 is Faster R-CNN multitask vehicle driving in reverse vision-based detection flow charts.

Embodiment

The invention will be further described below in conjunction with the accompanying drawings.

1~Fig. 5 of reference picture, a kind of multitask vehicle driving in reverse vision-based detection system based on depth convolutional neural networks System, including video camera, traffic Cloud Server and road vehicle reverse driving detecting system on urban road；

Described road vehicle reverse driving detecting system includes road and direction of traffic customized module, is based on FasterR-CNN vehicles and yellow line detection module, optical flow method direction of vehicle movement detection module, vehicle driving in reverse judge mould Block, vehicle type recognition module, License Plate and identification module and report on infringement of regulations generation module；Linking between each processing module As shown in Figure 5；

Described road and direction of traffic customized module are used for the track for customizing the road in camera field of view, specific practice It is that virtual track is drawn according to the lane on real road and direction of traffic in video image, virtual lane markings go out vehicle Travel direction；Here key be will mark go out the direction of traffic attribute that road is got on the bus；

In order to reduce the track direction of traffic customization link of user, a kind of preferable scheme is utilized in system prerun rank Section learns to obtain the direction of traffic in track automatically by self study mode；

Described is used to detect all cars in video image based on Faster R-CNN vehicles and yellow line detection module And road on yellow line, specific practice is the motor vehicles and Huang gone out on road using depth convolutional neural networks Fast Segmentation Line simultaneously provides these vehicles and yellow line spatial positional information shared on road；

Here, i is anchor index, p_iIt is the prediction probability that anchor is the i-th target, if anchor is Just, GT labelsIt is exactly 1, if anchor is negative,It is exactly 0；t_iIt is a vector, represents 4 parameters of the bounding box of prediction Change coordinate,It is the coordinate vector of GT bounding boxs corresponding with positive anchor；λ is a balance weight, here λ=10, N_clsIt is The normalized value of cls items is mini-batch size, here N_cls=256, N_regThe normalized value for being reg items is anchor positions The quantity put, N_reg=2,400；Classification Loss function L_clsTwo classifications, i.e. motor vehicles target and non power driven vehicle target Lost with the logarithm of yellow line target and road background：

For returning loss function L_reg, defined to minor function：

In formula, smooth_L1For smooth L₁Loss function, x are variable；

Because object of interest in the present invention is various motor vehicles and yellow line, i.e. object of interest, hereinafter referred to as RoI, will be various motor-driven in order to position and be partitioned into the various RoI on road, it is necessary in study and during training convolutional neural networks Yellow line and road background image on vehicle, road are put on corresponding label and are trained respectively；So in described base Motor vehicles and yellow line object of interest can be just partitioned into automatically in Faster R-CNN vehicles and the processing of yellow line detection module；

Described optical flow method direction of vehicle movement detection module is used to detect road vehicle travel direction；When road field When vehicle in scape corresponds to two dimensional image plane motion, these vehicles are formed moving in the projection of two dimensional image plane, The flowing that this motion is showed with plane of delineation luminance patterns is known as light stream.Optical flow method is that movement sequence image is carried out One important method of analysis, the movable information of Vehicle Object target in image is included in light stream；

V=u+d=[u_x+d_x u_y+d_y]^T (7)

The average value of the light stream vectors of certain vehicle inframe is calculatedAfterwards, if the value is more than a certain threshold value T, T=0.5； Just using the vehicle frame as moving vehicle, and the direction using the direction of the light stream vectors of the vehicle frame as moving vehicle；

Described vehicle driving in reverse determination module is used for the reverse driving behavior for the vehicle for identifying and judging road, inverse To traveling behavior, there is two ways：A kind of reverse driving behavior is the reverse row on the road of defined driveway travel directions Sail, for the judgement of the reverse driving behavior, first by the direction of traffic in the track that the system prerun stage obtains, according to The light stream vectors of the vehicle of road were calculated in described optical flow method direction of vehicle movement detection module, if the driving in track Angle between direction and the excessively light stream vectors of the vehicle of road is more than threshold value T_DIt is doubtful reverse driving behavior with regard to preliminary judgement, T_D=150 °；

Described Faster R-CNN are the depth convolutional neural networks of a multitask, as shown in figure 4, multitask embodies In detection positioning, the identification of vehicles peccancy type, the positioning of vehicles peccancy car plate and the identification to road vehicle and yellow line, These detection fixation and recognitions share the convolutional layer of same depth convolutional neural networks.

The foregoing is only the preferable implementation example of the present invention, be not intended to limit the invention, it is all in spirit of the invention and Within principle, any modification, equivalent substitution and improvements made etc., it should be included in the scope of the protection.

Claims

A kind of 1. multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks, it is characterised in that：Bag Include video camera, traffic Cloud Server and the road vehicle reverse driving detecting system on urban road；

Described video camera is used to obtain video data on each road in city, configures in the top of road, by network by road Vedio data on road is transferred to described traffic Cloud Server；

Described traffic Cloud Server is used to receive the road video data obtained from described video camera, and is submitted to Described road vehicle reverse driving detecting system carries out peccancy detection and to vehicles peccancy identification, finally by testing result Candid photograph image in merging scene automatically generates peccancy detection report and is stored in Cloud Server；

Described road vehicle reverse driving detecting system includes road and direction of traffic customized module, based on Faster R-CNN Vehicle and yellow line detection module, optical flow method direction of vehicle movement detection module, vehicle driving in reverse determination module, type of vehicle are known Other module, License Plate and identification module and report on infringement of regulations generation module；

Described road and direction of traffic customized module are used for the track for customizing the road in camera field of view, in video image Virtual track is drawn according to the lane on real road and direction of traffic, virtual lane markings go out the travel direction of vehicle；

Described Faster R-CNN are the depth convolutional neural networks of a multitask, are realized first to road in the network The detection positioning of vehicle and yellow line on road, then the type of vehicles peccancy is identified, then to the car plate of vehicles peccancy Positioned and identified, detection fixation and recognition shares the convolutional layer of same depth convolutional neural networks.
2. the multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks as claimed in claim 1, It is characterized in that：Described is used to detect owning in video image based on Faster R-CNN vehicles and yellow line detection module Yellow line on vehicle and road, specific practice be the motor vehicles that are gone out on road using depth convolutional neural networks Fast Segmentation and Yellow line simultaneously provides these vehicles and yellow line spatial positional information shared on road；

The motor vehicle segmentation and positioning used is made up of two models, and a model is the selective search network for generating RoI；Separately One model is Faster R-CNN motor vehicles targets and yellow line detection network；

Described selective search network, i.e. RPN；RPN networks export rectangular target using any scalogram picture as input The set of Suggestion box, each frame include 4 position coordinates variables and a score；The target of described target Suggestion box refers to Motor vehicles object and yellow line object；

It is the estimated probability of target/non-targeted to each Suggestion box, is the classification layer realized with the softmax layers of two classification；K Suggestion box is parameterized by the corresponding k Suggestion box for being referred to as anchor；

Each anchor is centered on current sliding window mouth center, and a kind of corresponding yardstick and length-width ratio, uses 3 kinds of yardsticks and 3 Kind length-width ratio, so just has k=9 anchor in each sliding position；

In order to train RPN networks, a binary label is distributed to each anchor, is to mark the anchor with this Target；Then distribute positive label and give this two class anchor：(I) have with some real target bounding box, i.e. Ground Truth, GT The ratio between highest IoU, i.e. Interse-ction-over-Union, common factor union, overlapping anchor；(II) with any GT bags Enclosing box has the overlapping anchor of the IoU more than 0.7；Notice that a GT bounding box may distribute positive label to multiple anchor； The IoU ratios that the negative label of distribution gives all GT bounding boxs are below 0.3 anchor；Anon-normal non-negative anchor is to training mesh No any effect is marked, then is abandoned；

The multitask loss in Faster R-CNN is followed, minimizes object function；The loss function of one image is defined as：

<mrow> <mi>L</mi> <mrow> <mo>(</mo> <mo>{</mo> <msub> <mi>p</mi> <mi>i</mi> </msub> <mo>}</mo> <mo>,</mo> <mo>{</mo> <msub> <mi>t</mi> <mi>i</mi> </msub> <mo>}</mo> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mn>1</mn> <msub> <mi>N</mi> <mrow> <mi>c</mi> <mi>k</mi> </mrow> </msub> </mfrac> <munder> <mo>&Sigma;</mo> <mi>i</mi> </munder> <msub> <mi>L</mi> <mrow> <mi>c</mi> <mi>l</mi> <mi>s</mi> </mrow> </msub> <mrow> <mo>(</mo> <msub> <mi>p</mi> <mi>i</mi> </msub> <mo>,</mo> <msubsup> <mi>p</mi> <mi>i</mi> <mo>*</mo> </msubsup> <mo>)</mo> </mrow> <mo>+</mo> <mi>&lambda;</mi> <mfrac> <mn>1</mn> <msub> <mi>N</mi> <mrow> <mi>r</mi> <mi>e</mi> <mi>g</mi> </mrow> </msub> </mfrac> <munder> <mo>&Sigma;</mo> <mi>i</mi> </munder> <msubsup> <mi>p</mi> <mi>i</mi> <mo>*</mo> </msubsup> <msub> <mi>L</mi> <mrow> <mi>r</mi> <mi>e</mi> <mi>g</mi> </mrow> </msub> <mrow> <mo>(</mo> <msub> <mi>t</mi> <mi>i</mi> </msub> <mo>,</mo> <msubsup> <mi>t</mi> <mi>i</mi> <mo>*</mo> </msubsup> <mo>)</mo> </mrow> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>)</mo> </mrow> </mrow>

Here, i is anchor index, p_iIt is the prediction probability that anchor is the i-th target, if anchor is just, GT marks LabelIt is exactly 1, if anchor is negative,It is exactly 0；t_iIt is a vector, represents 4 parametrization coordinates of the bounding box of prediction,It is the coordinate vector of GT bounding boxs corresponding with positive anchor；λ is a balance weight, N_clsThe normalized value for being cls items is Mini-batch size N_regIt is that the normalized value of reg items is the quantity of anchor positions；Classification Loss function L_clsIt is two The logarithm of classification, i.e. motor vehicles target and non power driven vehicle target and yellow line target and road background loses：

<mrow> <msub> <mi>L</mi> <mrow> <mi>c</mi> <mi>l</mi> <mi>s</mi> </mrow> </msub> <mrow> <mo>(</mo> <msub> <mi>p</mi> <mi>i</mi> </msub> <mo>,</mo> <msubsup> <mi>p</mi> <mi>i</mi> <mo>*</mo> </msubsup> <mo>)</mo> </mrow> <mo>=</mo> <mo>-</mo> <mi>l</mi> <mi>o</mi> <mi>g</mi> <mo>&lsqb;</mo> <msubsup> <mi>p</mi> <mi>i</mi> <mo>*</mo> </msubsup> <msub> <mi>p</mi> <mi>i</mi> </msub> <mo>+</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>-</mo> <msubsup> <mi>p</mi> <mi>i</mi> <mo>*</mo> </msubsup> <mo>)</mo> </mrow> <mrow> <mo>(</mo> <mn>1</mn> <mo>-</mo> <msub> <mi>p</mi> <mi>i</mi> </msub> <mo>)</mo> </mrow> <mo>&rsqb;</mo> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>2</mn> <mo>)</mo> </mrow> </mrow>

In formula, L_clsFor Classification Loss function, P_iIt is the prediction probability of the i-th target for anchor；P_i ^*For real target bounding box The prediction probability of i-th target；

For returning loss function L_reg, defined to minor function：

<mrow> <msub> <mi>L</mi> <mrow> <mi>r</mi> <mi>e</mi> <mi>g</mi> </mrow> </msub> <mo>=</mo> <mrow> <mo>(</mo> <msub> <mi>t</mi> <mi>i</mi> </msub> <mo>,</mo> <msubsup> <mi>t</mi> <mi>i</mi> <mo>*</mo> </msubsup> <mo>)</mo> </mrow> <mo>=</mo> <mi>R</mi> <mrow> <mo>(</mo> <msub> <mi>t</mi> <mi>i</mi> </msub> <mo>-</mo> <msubsup> <mi>t</mi> <mi>i</mi> <mo>*</mo> </msubsup> <mo>)</mo> </mrow> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>3</mn> <mo>)</mo> </mrow> </mrow>

In formula, L_regTo return loss function, R is the loss function of robust, and smooth L are calculated with formula (4)₁；

<mrow> <msub> <mi>smooth</mi> <mrow> <mi>L</mi> <mn>1</mn> </mrow> </msub> <mrow> <mo>(</mo> <mi>x</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mrow> <mn>0.5</mn> <msup> <mi>x</mi> <mn>2</mn> </msup> </mrow> </mtd> <mtd> <mtable> <mtr> <mtd> <mrow> <mi>i</mi> <mi>f</mi> </mrow> </mtd> <mtd> <mrow> <mrow> <mo>|</mo> <mi>x</mi> <mo>|</mo> </mrow> <mo><</mo> <mn>1</mn> </mrow> </mtd> </mtr> </mtable> </mtd> </mtr> <mtr> <mtd> <mrow> <mrow> <mo>|</mo> <mi>x</mi> <mo>|</mo> </mrow> <mo>-</mo> <mn>0.5</mn> </mrow> </mtd> <mtd> <mrow> <mi>o</mi> <mi>t</mi> <mi>h</mi> <mi>e</mi> <mi>r</mi> <mi>w</mi> <mi>i</mi> <mi>s</mi> <mi>e</mi> <mo>,</mo> </mrow> </mtd> </mtr> </mtable> </mfenced> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>4</mn> <mo>)</mo> </mrow> </mrow>

In formula, smooth_L1For smooth L₁Loss function, x are variable；

Faster R-CNN networks, characteristic pattern can be obtained after depth convolutional neural networks in input picture, according to feature Figure and RPN networks can then obtain corresponding RoIs, finally then pass through RoI ponds layer；Wherein RoI, i.e. area-of-interest, refer to Be exactly yellow line zone on the region and road of motor vehicle；

For Faster R-CNN networks, input is N number of Feature Mapping and R RoI；N number of Feature Mapping comes from last Convolutional layer, the size of each Feature Mapping is w × h × c；

Each RoI is a tuple (n, r, c, h, w), wherein, n is the index of Feature Mapping, n ∈ (0,1,2 ..., N-1), r, C is top left co-ordinate, and h, w are height and width respectively；

Export the Feature Mapping then obtained by maximum pond；RoI in artwork is mapped with the block in characteristic pattern；By feature Figure down-sampling is fixed size, is then passed to full connection again.
3. the multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks as claimed in claim 2, It is characterized in that：The selective search network and Fast R-CNN are all stand-alone trainings, using 4 step training algorithms, are passed through Alternative optimization learns shared feature；The first step, according to above-mentioned training RPN, at the beginning of model of the network with ImageNet pre-training Beginningization, and end-to-end finely tune suggests task for region；Second step, the Suggestion box generated using the RPN of the first step, by Fast R-CNN train one individually detection network, this detection network be equally by the model initialization of ImageNet pre-training, At this time two networks are also without shared convolutional layer；3rd step, trained with detection netinit RPN, but fixed shared volume Lamination, and only finely tune the exclusive layers of RPN, present two network share convolutional layers；4th step, shared convolutional layer is kept to consolidate It is fixed, fine setting Fast R-CNN fc, i.e., full articulamentum；So, two network share identical convolutional layers, form one it is unified Network；

By the processing of above-mentioned two network, yellow line on the motor vehicles and road in a frame video image is detected and to it Size and locus confined, that is, obtained size and the locus of yellow line on vehicle and road, its r_v, c_vIt is the top left co-ordinate of vehicle in the picture, h_v,w_vProjected size of the vehicle in the plane of delineation respectively, i.e. vehicle image Height and width；Its r_y,c_yIt is the top left co-ordinate of yellow line in the picture on road, h_y,w_yIt is the yellow line on road respectively in image The height and width of the projected size of plane, i.e. yellow line image.
4. the inspection of the multitask vehicle driving in reverse vision based on depth convolutional neural networks as described in one of claims 1 to 3 Examining system, it is characterised in that：Described optical flow method direction of vehicle movement detection module is used to detect road vehicle traveling side To；When the vehicle in road scene correspond to two dimensional image plane move when, these vehicles two dimensional image plane projection just Motion is formd, the flowing that this motion is showed with plane of delineation luminance patterns is known as light stream；Optical flow method is to motion The important method that sequence image is analyzed, the movable information of Vehicle Object target in image is included in light stream, that is, is moved Vector；

Using the sparse iterative method of Lucas-Kanade light streams based on pyramid model, it is assumed that image I size is n_x×n_y, it is fixed Adopted I⁰Represent the 0th tomographic image, the 0th tomographic image is rate highest image, i.e. original image respectively, this tomographic image it is wide and a height ofWith Then we describe pyramidal representation with a kind of recursive mode：We pass through I^L-1 To calculate I^L, L=1,2 ..., I^L-1Represent the image of pyramid L-1 layers, I^LRepresent the image of pyramid L layers, it is assumed that image I^L-1 It is wide and a height ofWithSo image I^LIt is expressed as

<mfenced open = "" close = ""> <mtable> <mtr> <mtd> <mrow> <msup> <mi>I</mi> <mi>L</mi> </msup> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mn>1</mn> <mn>4</mn> </mfrac> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mn>2</mn> <mi>x</mi> <mo>,</mo> <mn>2</mn> <mi>y</mi> <mo>)</mo> </mrow> <mo>+</mo> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mfrac> <mn>1</mn> <mn>8</mn> </mfrac> <mrow> <mo>(</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>-</mo> <mn>1</mn> <mo>,</mo> <mn>2</mn> <mi>y</mi> </mrow> <mo>)</mo> <mo>+</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>+</mo> <mn>1</mn> <mo>,</mo> <mn>2</mn> <mi>y</mi> </mrow> <mo>)</mo> <mo>+</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>,</mo> <mn>2</mn> <mi>y</mi> <mo>-</mo> <mn>1</mn> </mrow> <mo>)</mo> <mo>+</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>,</mo> <mn>2</mn> <mi>y</mi> <mo>+</mo> <mn>1</mn> </mrow> <mo>)</mo> <mo>)</mo> </mrow> <mo>+</mo> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mfrac> <mn>1</mn> <mn>16</mn> </mfrac> <mrow> <mo>(</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>-</mo> <mn>1</mn> <mo>,</mo> <mn>2</mn> <mi>y</mi> <mo>-</mo> <mn>1</mn> </mrow> <mo>)</mo> <mo>+</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>+</mo> <mn>1</mn> <mo>,</mo> <mn>2</mn> <mi>y</mi> <mo>+</mo> <mn>1</mn> </mrow> <mo>)</mo> <mo>+</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>-</mo> <mn>1</mn> <mo>,</mo> <mn>2</mn> <mi>y</mi> <mo>+</mo> <mn>1</mn> </mrow> <mo>)</mo> <mo>,</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mo>(</mo> <mrow> <mn>2</mn> <mi>x</mi> <mo>+</mo> <mn>1</mn> <mo>,</mo> <mn>2</mn> <mi>y</mi> <mo>-</mo> <mn>1</mn> </mrow> <mo>)</mo> <mo>)</mo> </mrow> </mrow> </mtd> </mtr> </mtable> </mfenced>

By imageThe value of boundary point be defined as follows,

<mrow> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mo>-</mo> <mn>1</mn> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mover> <mo>=</mo> <mo>&CenterDot;</mo> </mover> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mn>0</mn> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> </mrow>

<mrow> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mo>-</mo> <mn>1</mn> <mo>)</mo> </mrow> <mover> <mo>=</mo> <mo>&CenterDot;</mo> </mover> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mn>0</mn> <mo>)</mo> </mrow> </mrow>

<mrow> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <msubsup> <mi>n</mi> <mi>x</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mover> <mo>=</mo> <mo>&CenterDot;</mo> </mover> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <msubsup> <mi>n</mi> <mi>x</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>-</mo> <mn>1</mn> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> </mrow>

<mrow> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <msubsup> <mi>n</mi> <mi>y</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>)</mo> </mrow> <mover> <mo>=</mo> <mo>&CenterDot;</mo> </mover> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <mi>y</mi> <mo>,</mo> <msubsup> <mi>n</mi> <mi>y</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>-</mo> <mn>1</mn> <mo>)</mo> </mrow> </mrow>

<mrow> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <msubsup> <mi>n</mi> <mi>x</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>,</mo> <msubsup> <mi>n</mi> <mi>y</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>)</mo> </mrow> <mover> <mo>=</mo> <mo>&CenterDot;</mo> </mover> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mrow> <mo>(</mo> <msubsup> <mi>n</mi> <mi>x</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>-</mo> <mn>1</mn> <mo>,</mo> <msubsup> <mi>n</mi> <mi>y</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>-</mo> <mn>1</mn> <mo>)</mo> </mrow> </mrow>

The point that formula (5) defines must is fulfilled for conditionTherefore image I^LWidthAnd heightNeed to meet formula (6),

<mrow> <msubsup> <mi>n</mi> <mi>x</mi> <mi>L</mi> </msubsup> <mo>&le;</mo> <mfrac> <mrow> <msubsup> <mi>n</mi> <mi>x</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>+</mo> <mn>1</mn> </mrow> <mn>2</mn> </mfrac> </mrow>

<mrow> <msubsup> <mi>n</mi> <mi>y</mi> <mi>L</mi> </msubsup> <mo>&le;</mo> <mfrac> <mrow> <msubsup> <mi>n</mi> <mi>y</mi> <mrow> <mi>L</mi> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <mo>+</mo> <mn>1</mn> </mrow> <mn>2</mn> </mfrac> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>6</mn> <mo>)</mo> </mrow> </mrow>

Image I pyramid model { I is built by formula (5) and (6)_LL=0 ..., L_m, L_mFor the height of pyramid model Degree；

LK optical flow computation methods based on pyramid model, first the top k layers search characteristics point in image pyramid model Match point, then kth -1 layer of the initial estimate in image pyramid model using the result of calculation of k layers as k-1 layers search Rope match point, go round and begin again and iterate to the 0th layer of image pyramid model always, so as to which the light stream of this feature point be calculated；

The detection target of optical flow method is：In front and rear two field pictures I and J, for image I some pixel u, in image J Its match point v=u+d is found, or finds out its offset vector d, is calculated with formula (7)；

V=u+d=[u_x+d_x u_y+d_y]^T (7)

In formula, u is some pixel in image I, and v is matched pixel in image J, and d is between the two inclined The amount of shifting to；

First, image I and J pyramid model { I are established_LL=0 ..., L_m{ J_LL=0 ..., L_m；Then pixel u is calculated The position in each pyramidal layers of image IL=0 ..., L_m；Then by a search window image J pyramid Model highest tomographic image IL_mMiddle calculating uL_mMatch point vL_m, and calculate offset vector d^Lm；

Next we describe the optical flow method based on pyramid model with the mode of iteration；Assuming that pyramid model is known The offset vector d of L+1 layers^L+1, then by 2d^L+1As the initial value of L layers, with the match point vL for nearby searching for L layers；Enter And obtain the offset vector d of L layers^L；

By each layer of offset vector d of iterative calculation^LAfter (L=0 ..., Lm), the final light stream of the pixel is

<mrow> <mi>d</mi> <mo>=</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>L</mi> <mo>=</mo> <mn>0</mn> </mrow> <msub> <mi>L</mi> <mi>m</mi> </msub> </munderover> <msup> <mn>2</mn> <mi>L</mi> </msup> <msup> <mi>d</mi> <mi>L</mi> </msup> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>8</mn> <mo>)</mo> </mrow> </mrow>

In formula, d be a certain pixel light stream value, d^LFor a certain pixel L layers light stream value；

After the light stream vectors value of each feature pixel in obtaining image, detected according to described vehicle detection unit on road Motor vehicles and shared spatial positional information on road, i.e., the frame of each vehicle has been obtained in two dimensional image plane, Each frame has four data representations, the position r in the upper left corner, c and length and width h, w；Here all spies of each inframe are calculated The average value of sign point light stream vectors, is calculated with formula (9),

<mrow> <mover> <mi>d</mi> <mo>&OverBar;</mo> </mover> <mo>=</mo> <munderover> <mo>&Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>0</mn> </mrow> <mi>n</mi> </munderover> <msub> <mi>d</mi> <mi>i</mi> </msub> <mo>/</mo> <mi>n</mi> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>9</mn> <mo>)</mo> </mrow> </mrow>

In formula,For the average value of the light stream vectors of certain vehicle inframe, d_iFor the light stream of a certain feature pixel of certain vehicle inframe Vector value, n are the quantity of the feature pixel of certain vehicle inframe；

The average value of the light stream vectors of certain vehicle inframe is calculatedAfterwards, if the value is more than a certain threshold value T, T=0.5, just will The vehicle frame is as moving vehicle, and the travel direction using the direction of the light stream vectors of the vehicle frame as moving vehicle.
5. the inspection of the multitask vehicle driving in reverse vision based on depth convolutional neural networks as described in one of claims 1 to 3 Examining system, it is characterised in that：In described road and direction of traffic customized module, pass through self-study using in the system prerun stage Habit mode learns to obtain the direction of traffic in track automatically；First, examined by described based on Faster R-CNN vehicles and yellow line Survey module detects all vehicles in video image, then, passes through described optical flow method direction of vehicle movement detection module meter The light stream vectors for obtaining detecting these vehicles are calculated, finally, these light stream vectors values is done into vector and added, this vector adds obtained knot Fruit can serve as the direction of traffic in track；The time in system prerun stage according to cross road number of vehicles depending on, here It is 100 by the number of vehicles for crossing road, after this number is reached, the customization work of the direction of traffic of road terminates automatically.
6. the inspection of the multitask vehicle driving in reverse vision based on depth convolutional neural networks as described in one of claims 1 to 3 Examining system, it is characterised in that：Described vehicle driving in reverse determination module is used for the reverse of the vehicle for identifying and judging road Traveling behavior, there is two ways for reverse driving behavior：A kind of reverse driving behavior is in defined driveway travel directions Reverse driving on road, for the judgement of the reverse driving behavior, first by the track that the system prerun stage obtains Direction of traffic, the light stream vectors of the vehicle of road were calculated according to described optical flow method direction of vehicle movement detection module, If the angle between the direction of traffic in track and the excessively light stream vectors of the vehicle of road is more than threshold value T_DIt is doubtful with regard to preliminary judgement Like reverse driving behavior, T_D=150 °；

Another reverse driving behavior is to cross yellow line to enter inverted running track, for the judgement of this reverse behavior, first It is whether to have yellow line on road to be detected, also judges whether vehicle pushes down yellow line or cross yellow line if yellow line； When occurring due to such case, on road yellow line some parts by occlusion, so need by algorithm by yellow line part also Original comes out；With the yellow line on Hough transform fitting road, the angle that yellow line travels through is limited in [- 5 ° ,+5 °]；Tried to achieve in calculating On road after yellow line, then detect that Vehicle Object is based on Faster R-CNN vehicles and yellow line detection module according to described It is no to have overlapping and covering phenomenon with yellow line, if being doubtful reverse driving behavior with regard to preliminary judgement；

Using continuous candid photograph and identification technology, a sign of flag is set, this marked when having detected reverse driving behavior Will Flag adds 1, when first time preliminary judgement is doubtful reverse driving behavior, Flag=1, captures piece image；For the second time still When so having detected reverse driving behavior, Flag=2, the second width image is captured；Third time has still detected reverse driving row For when, Flag=3, capture the 3rd width image；Flag=3 situation is it is confirmed that it is reverse driving behavior to be；Finally by this three width Image is stored in in the file of the name of time at that time；If do not detected reverse driving behavior, Flag=0.
7. the inspection of the multitask vehicle driving in reverse vision based on depth convolutional neural networks as described in one of claims 1 to 3 Examining system, it is characterised in that：Described vehicle type recognition module is used to the type of vehicle of reverse driving be identified, to carry The level of high vehicles peccancy identification capability and automatic business processing traffic offence event；Here multitask depth convolution is used Nerual network technique, it is described based on the Faster R-CNN networks of Faster R-CNN vehicles and yellow line detection module in again A center loss function and a softmax loss function are added, realizes center loss function and softmax loss functions Joint-monitoring, export to obtain the brand of vehicles peccancy, series, body color information by softmax；Described type of vehicle is known Other module uses non real-time calculation, after described vehicle driving in reverse determination module is determined with reverse driving vehicle, Start a thread, read the image that vehicles peccancy is captured with three width interior in the file of time name, it is carried out respectively Vehicle type recognition.
8. the inspection of the multitask vehicle driving in reverse vision based on depth convolutional neural networks as described in one of claims 1 to 3 Examining system, it is characterised in that：Described License Plate and identification module are used to the car plate of the vehicle of reverse driving be identified, Further to improve the level of vehicles peccancy identification capability and automatic business processing traffic offence event；When described vehicle After reverse driving determination module is determined with reverse driving vehicle, start a thread, read with interior in the file of time name Three width capture vehicles peccancy image, then respectively to three width capture vehicles peccancy image on car plate position determine Position, obtains the image of car plate, then license plate image is identified, identify the license plate number of vehicles peccancy.
9. the inspection of the multitask vehicle driving in reverse vision based on depth convolutional neural networks as described in one of claims 1 to 3 Examining system, it is characterised in that：Described report on infringement of regulations generation module is used to automatically generate the report for punishing vehicles peccancy, The brand of the vehicles peccancy obtained according to described vehicle type recognition resume module, series, body color information, described car The license plate number of vehicles peccancy that board positions and identification module obtains, then according to these message reference vehicle managements vehicle step on Remember database, whether the vehicle identity information of matching identification and the vehicle identity information registered are consistent, automatic raw if consistent The report punished into vehicles peccancy；If inconsistent, the information is pushed to administrative staff, further confirmed；Report Content in announcement includes：Image, vehicle identity information, place violating the regulations and the time that three width are captured when violating the regulations.