CN109784306B - Intelligent parking management method and system based on deep learning - Google Patents

Abstract

The invention discloses an intelligent parking management method and system based on deep learning. Aiming at the requirement of intelligent parking lot management, a convolutional neural network model is designed based on a Darknet frame to position a vehicle, the predicted vehicle position and the predicted parking space are subjected to overlapping degree calculation, the parking state of the parking space is monitored in real time, and parking cost statistics is carried out according to the parking space state. The invention realizes intelligent and automatic monitoring of parking state and parking cost by applying the deep learning method to the parking resource management, relieves the urban traffic congestion, effectively standardizes the use of parking resources, and simultaneously leads the management of the parking resources to be more convenient and intelligent, and liberates manpower.

Description

A method and system for intelligent parking management based on deep learning

technical field

The invention relates to the technical field of intelligent parking management, in particular to a deep learning-based intelligent parking management method and system.

Background technique

In recent years, with the growth of private cars, large, medium and small cities are faced with the dilemma of "more cars and fewer seats". At the same time, the current situation of backward management of parking resources has also made this dilemma more and more serious. The traditional management of parking resources is managed in a manual mode, which is not only inefficient, but also extremely labor-intensive.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide an intelligent parking management method and system based on deep learning, and apply the deep learning method to parking resource management, so as to solve the problems of low efficiency and extremely labor-consuming traditional manual management of parking resources.

For achieving the above object, the present invention provides the following scheme:

An intelligent parking management method based on deep learning, the method includes:

Get the video frame captured by the parking lot camera;

Load the trained convolutional neural network model;

Inputting the video frame into the trained convolutional neural network model, and outputting vehicle position prediction frame information; the vehicle position prediction frame information includes the center coordinates, width and height of the vehicle position prediction frame;

obtaining parking space frame information; the parking space frame information includes the coordinates of the upper left corner and the lower right corner of the parking space frame;

Determine the current parking space state according to the vehicle position prediction frame information and the parking space frame position information;

The parking fee is determined according to the current parking space state.

Optionally, before loading the trained convolutional neural network model, the method further includes:

A convolutional neural network model is established based on the Darknet framework and the YOLO algorithm; the convolutional neural network model includes a feature extractor and a detector;

The VOC data set is used to train and adjust parameters of the convolutional neural network model to generate a trained convolutional neural network model.

Optionally, the determining the current parking space state according to the vehicle position prediction frame information and the parking space frame position information specifically includes:

Calculate the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame according to the vehicle position prediction frame information;

The position of the overlapping area between the vehicle position prediction frame and the parking space frame is calculated according to the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame, and the coordinates of the upper left corner and the lower right corner of the parking space frame; the overlapping area position Including the coordinates of the upper left corner and the lower right corner of the overlapping area of the vehicle position prediction frame and the parking space frame;

Calculate the overlapping area area according to the overlapping area position;

Calculate the area of the parking space frame according to the position of the parking space frame;

Calculate the ratio of the area of the overlapping area to the area of the parking space frame as the confidence level of the vehicle;

Determine whether the confidence of the vehicle is greater than or equal to a confidence threshold, and obtain a first judgment result;

If the first judgment result is that the confidence of the vehicle is greater than or equal to the confidence threshold, determine that the current parking space state is an occupied state;

If the confidence level of the vehicle is less than the confidence level threshold, it is determined that the current parking space state is an empty state.

Optionally, the determining the parking fee according to the current parking space state specifically includes:

Obtain the starting parking time when the current parking space state changes from an empty state to an occupied state;

Obtain the ending parking time when the current parking space state changes from the occupied state to the empty state;

Calculate the parking time of the vehicle according to the start parking time and the end parking time;

The parking fee is determined according to the parking time.

An intelligent parking management system based on deep learning, the system includes:

The video capture module is used to obtain the video frames captured by the parking lot camera;

The model loading module is used to load the trained convolutional neural network model;

The vehicle position prediction module is used to input the video frame into the trained convolutional neural network model, and output the vehicle position prediction frame information; the vehicle position prediction frame information includes the center coordinates of the vehicle position prediction frame and the width and height of the frame. ;

a parking space frame acquisition module, used for obtaining parking space frame information; the parking space frame information includes the coordinates of the upper left corner and the lower right corner of the parking space frame;

The current parking space state judgment module is configured to determine the current parking space state according to the vehicle position prediction frame information and the parking space frame position information;

The parking fee determination module is used for determining the parking fee according to the current parking space state.

Optionally, the system further includes a model establishment module, and the model establishment module specifically includes:

A model establishment unit for establishing a convolutional neural network model based on the Darknet framework and the YOLO algorithm; the convolutional neural network model includes a feature extractor and a detector;

The model training unit is used to train and adjust parameters of the convolutional neural network model by using the VOC data set to generate a trained convolutional neural network model.

Optionally, the current parking space state judgment module specifically includes:

a vehicle position calculation unit, configured to calculate the upper left corner coordinate and the lower right corner coordinate of the vehicle position prediction frame according to the vehicle position prediction frame information;

an overlapping area position calculation unit, configured to calculate the overlap between the vehicle position prediction frame and the parking space frame according to the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame and the coordinates of the upper left corner and the lower right corner of the parking space frame area position; the overlapping area position includes the coordinates of the upper left corner and the lower right corner of the overlapping area of the vehicle position prediction frame and the parking space frame;

an overlapping area area calculation unit, configured to calculate the overlapping area area according to the overlapping area position;

a parking space frame area calculation unit, configured to calculate the parking space frame area according to the position of the parking space frame;

a confidence degree calculation unit, configured to calculate the ratio of the area of the overlapping area to the area of the parking space frame as the confidence degree of the vehicle;

a confidence level judgment unit, configured to judge whether the confidence level of the vehicle is greater than or equal to a confidence level threshold, and obtain a first judgment result;

an occupied state judgment unit, configured to determine that the current parking space state is an occupied state if the first judgment result is that the confidence level of the vehicle is greater than or equal to a confidence level threshold;

An empty state judging unit, configured to determine that the current parking space state is an empty state if the confidence level of the vehicle is less than the confidence level threshold.

Optionally, the parking fee determination module specifically includes:

a parking start time recording unit, used to obtain the start parking time when the current parking space state changes from an empty state to an occupied state;

an end parking time recording unit, used to obtain the end parking time when the current parking space state changes from an occupied state to an empty state;

a parking time calculation unit, configured to calculate the parking time of the vehicle according to the start parking time and the end parking time;

A parking fee calculation unit, configured to determine the parking fee according to the parking time.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The present invention provides an intelligent parking management method and system based on deep learning. According to the requirements of intelligent parking lot management, a convolutional neural network model is designed based on the Darknet framework to locate the vehicle, and the overlap degree of the predicted vehicle position and the parking space is calculated. , realize the real-time monitoring of the parking status on the parking space, and conduct the parking fee statistics according to the parking space status. By applying the deep learning method to the management of parking resources, the present invention realizes intelligent and automatic monitoring of parking status and parking fees, which not only alleviates urban traffic congestion, effectively regulates the use of parking resources, but also enables the management of parking resources. More convenient and intelligent, liberating manpower.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

Fig. 1 is the flow chart of the intelligent parking management method based on deep learning provided by the present invention;

Fig. 2 is the data flow diagram of the ARM end provided by the present invention;

Fig. 3 is the data processing flow chart of the server end provided by the present invention;

4 is a network structure diagram of a feature extractor and a detector in the convolutional neural network model provided by the present invention;

FIG. 5 is a structural diagram of an intelligent parking management system based on deep learning provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The invention relates to related technical fields such as deep learning, computer vision, Darknet framework, real-time communication, image and video transmission, multi-thread parallelism, ARM development, C++ graphical user interface application development, etc., in particular to the related technical fields in deep learning and computer vision Design and training of deep learning neural networks for object classification, localization, detection, and fusion of parking spaces and detected objects. The purpose of the present invention is to provide an intelligent parking management method and system based on deep learning. The deep learning method is applied to parking resource management. The application of deep learning technology in computer vision makes machine vision and human vision closer, and even In some aspects, its performance has surpassed that of human beings. Using this technology to track and locate vehicles in real time can effectively conduct real-time statistics on vehicle entry and exit, so as to solve the problem of inefficient and extremely labor-intensive traditional manual management of parking resources. .

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

The key issues addressed by the method and system of the present invention include:

1) Analyze the positioning of the system and the composition of the system;

2) How to transmit video images;

3) How to judge the status of the parking space;

4) How to carry out target positioning;

5) How to extract parking space information;

6) Calculate according to the method of 3) to monitor the parking space status in real time;

7) How the system is integrated.

Among the above seven main questions, questions 1) and 7) are the overall situation, and questions 2), 4) and 5) are the core. The present invention mainly solves these 7 problems in the following ways:

1. The method and system of the present invention are aimed at the unified management of existing parking resources that are difficult to manage, but it is difficult to monitor the parking lot through a camera, so every 3 to 5 parking spaces is monitored by a camera. The system consists of a front-end ARM to control the video capture and forwarding of the camera, and the background receiving server to process the data received from different cameras. Since there are multiple cameras to monitor multiple parking spaces, it is necessary to process the results of these different cameras. It is reflected on the real-time screen, so it is necessary to use multiple threads to transmit the data of different cameras to the respective real-time screen.

2. In view of the reliability of the TCP protocol, the present invention adopts TCP for video transmission in the system communication transmission part. The front-end to back-end connection selects the 4G communication module, and users can use the Internet to connect the front-end and back-end. Multiple cameras are connected to an ARM embedded development board to manage an area, and a frame of image obtained by each camera is added to the camera number information. Multi-threaded operation is performed on the ARM side, and the region information and camera information are added and integrated into a data packet, which is packaged and sent to the server.

3. The judgment of the parking space status needs to be determined by the position detection result of the car in the video and the position information of the parking space. After calculating the overlapping part of the target vehicle prediction frame and the parking space, as well as the overlapping ratio of the overlapping part and the parking space, the state of the parking space is judged.

4. The position detection of the car adopts the current deep learning technology, based on the Darknet framework, the idea in the YOLO algorithm and a 23-layer pre-training model, on this basis, the detector is designed, and the design of the detector is given in Figure 4; Then train and adjust parameters on the VOC data set until the optimization goal is met, and finally a trained convolutional neural network model is obtained. After the video frame is processed by the model, the information about the prediction frame of the target vehicle object in the video picture is obtained, and the obtained prediction frame of the vehicle is represented by four key parameters.

5. Through the parking space acquisition method provided by the invention, code is written to realize the UI interface of parameter adjustment, so that the system can be installed and debugged by the administrator himself through the manual.

6. Real-time detection of parking space status, and multi-threaded real-time display of multiple cameras.

7. System integration, which is to integrate the system platform of the previous designs in 1 to 6. All program modules have passed the unit test, and then the whole system is tested and debugged, and this process is repeated until the whole system runs stably.

According to the above inventive concept, the present invention proposes an intelligent parking management method and system based on deep learning.

FIG. 1 is a flowchart of a deep learning-based intelligent parking management method provided by the present invention. Referring to FIG. 1, the deep learning-based intelligent parking management method provided by the present invention specifically includes:

Step 101: Acquire the video frames collected by the parking lot cameras.

Since the parking lot needs to track the entry and exit of vehicles in real time to facilitate the statistics of parking space status in real time, it is necessary to transmit the video collected by the parking lot camera to the background server in time.

Since the transmission of the video captured by the camera needs to be stable and reliable, the present invention adopts the TCP (Transmission Control Protocol) transmission protocol; in order to meet the time requirement, the video frame captured by the camera is transmitted by 4G technology, which integrates 3G and WLAN, Capable of transferring data such as high-quality video, audio, and images quickly.

The parking lot generally uses multiple cameras to collect video from multiple parking spaces. The present invention sets each camera to monitor 4 parking spaces, so each camera needs a display screen and needs to be displayed in real time. Therefore, it is necessary to monitor each camera to determine the status and timing of the parking space, and display the monitored video data and their respective status and timing information on their corresponding real-time images through their respective threads.

The video captured by the camera is first captured by the ARM (Advanced RISC Machines) board, then numbered, and forwarded to the background server through the TCP protocol. Due to the limited memory on the ARM side, it is necessary to perform certain memory optimizations on the program during development to avoid memory overflow and crash when the ARM side processes images after grabbing them, causing unnecessary trouble. By numbering and monitoring multiple cameras, the video frames captured by each camera are marked and forwarded to the background server.

FIG. 2 is a data flow diagram of the ARM side provided by the present invention. Since the present invention uses one camera to monitor a limited area, multiple cameras are required for a parking lot to monitor different areas. Multiple cameras transmit the collected data frames to the ARM side, and the ARM side processes the data. Before receiving the data, the ARM side checks whether the space it has allocated for the task is enough, and then receives it if it is enough, otherwise the current frame is discarded. After encoding, the frame is forwarded to the backend server.

The final monitoring display for parking spaces is performed by a management interface developed with Qt. Qt is a cross-platform graphical user interface application development framework. It uses special code generation extensions and some macros in an object-oriented way to allow component programming, which greatly improves development efficiency.

The server side receives the video data forwarded from the ARM side, and puts each data into the network for prediction. The backend server puts the returned video into the trained convolutional neural network model. After processing through the network model, the vehicle can be detected in real time, and the ratio of the overlapping part of the parking space and the vehicle position to the parking space can be calculated. According to the result of this ratio calculation Whether the set threshold is met to judge whether the parking space is empty, and count the time when the parking space is not empty until the parking space is empty; when the parking space changes from empty to not empty, start timing until the parking space changes from not empty to empty. Stop when the state is empty Time, at this time calculate the parking fee that should be paid.

The vacancy and non-empty of parking spaces need to be displayed in real time, and the parking time also needs to be displayed in real time. Therefore, in addition to detecting each parking space, it is also necessary to display the detection status in real time, that is, it is necessary to analyze the parking spaces detected by all cameras and give the results; create a thread for each camera to collect the videos of all cameras uniformly information, and number each camera and the collected video information, output them in sequence, and display them on the management interface of the corresponding camera in real time.

However, although each data frame has been numbered on the ARM side, and the server in the background can know which camera the corresponding frame comes from, since the network model only needs video frames, not the camera's label information, after the network model processes a When you frame an image, you don't know which camera the image is from. Therefore, you first need to put the video frame collected by the camera into the buffer queue, and set a flag[n] tag array to record the frame processed in the current network. From which camera, initialize all elements in the flag to 0. When the data of the jth camera is processed in the network, the flag[j] is changed to 1. After the predicted value is obtained, the flag can be obtained from the flag. Which camera the video frame belongs to; at the same time, since the elements in the flag can only represent one camera, the flag needs to be locked, and the prediction process of the network model cannot be disturbed, so the prediction process of the network model also needs to be locked; network After the frame operation of the current camera is completed, it needs to unlock itself and then unlock the flag, so that the data in the buffer queue can continue to be detected in the network model, and the corresponding results are reflected in the corresponding camera for display.

FIG. 3 is a flow chart of data processing on the server side provided by the present invention. Before the server side receives the data from the ARM side, it first checks the space it has reserved for receiving video frames. If the current video frame can be stored, it will be stored, otherwise the current frame will be discarded. In order to reduce the performance dependence on machine equipment, a single frame enters a single network for calculation, and the calculation result is the vehicle position prediction frame. The background server of the present invention processes the vehicle position frame and the parking space frame and updates the parking space status.

Since multiple cameras are used to monitor parking spaces, each camera will independently output its own parking space status to the corresponding real-time screen, and the parking space corresponding to each camera will have a position initialization for the parking space. Therefore, after the predicted position of the car is obtained, the initialization information corresponding to the jth camera is found, and subsequent processing is performed. In order to enable the pictures of each camera to be independently displayed in their respective pictures in real time, the present invention uses multiple threads to process the information of different cameras, so as to achieve stable and reliable real-time monitoring of the pictures captured by multiple cameras.

Step 102: Load the trained convolutional neural network model.

The design of the convolutional neural network model used in the identification and detection of the vehicle of the present invention is realized based on the Darknet framework. Darknet is a lightweight deep learning framework, implemented in C language, supports CPU (Central Processing Unit, central processing unit) and GPU (Graphics Processing Unit, graphics processor), it does not have a powerful API (Application Programming Interface, application programming interface) ), but it is its lightweight implementation that makes it easier to use. It is also very convenient to use Darknet to design and train the network. In addition to the fact that the darknet environment is not so easy to build, it is extremely convenient to use it for network design and training. Since opencv (Open SourceComputer Vision Library, open source computer vision library) has officially supported the Draknet framework since 3.3.1, the model obtained after GPU training can be used on the CPU, reducing the requirements for using the machine.

The convolutional neural network model established by the present invention includes a feature extractor and a detector. Based on the Darknet framework and the target detection idea in algorithms such as YOLO (You Only Look Once), the detector is designed on the basis of the Darknet19 feature extractor. . The detector is mainly used to detect the position of the vehicle in the video. It adopts a top-down method to identify, locate and detect the target object on the features of different scales, so as to make the recognition of the target more accurate.

The detailed network structure design of the feature extractor and the detector in the convolutional neural network model of the present invention is shown in Figure 4 and Table 1 below:

Table 1 Convolutional Neural Network Model Network Structure

Type in Table 1 represents the layer type, Filters represents the number of convolution kernels (that is, the number of output channels), Size/Stride represents the size/stride of the Filter, Output represents the output, and Convolutional or conv in Table 1 and Figure 4 represents the volume Product, Maxpool or maxpool means maximum pooling, YOLO or yolo comes from the method proposed in the article "You Only Look Once: Unified, Real-Time Object Detection", there is no suitable Chinese expression, route means routing, upsample means upsampling, Concate means cascade.

Referring to Table 1 and Figure 4, the network design of the convolutional neural network model established by the present invention is mainly divided into two parts, a feature extraction part and a detection part. The feature extraction part adopts the Darknet19 feature extraction model, which is responsible for detecting the features in the video frame and providing the detection basis for the subsequent detection part; the detection part is mainly responsible for the detection of the feature map provided by the feature extraction part, and the final result is the The predicted box of the detected object, used to represent the location of the object.

Specifically, the feature extractor in the convolutional neural network model adopts Darknet19, and Darknet19 has a total of 23 layers, including 18 convolutional layers and 5 maximum pooling layers. The detector is designed to make predictions at two scales, with a total of 12 layers, of which two Yolo layers are used as output layers to detect at scales of 13x13 and 26x26, respectively. There are three convolutional layers and one yolo layer on the 13x13 scale; the yolo layer outputs the prediction results on the 13x13 scale. Then route the 23rd layer to the 28th layer for a convolution, up-sampling on the convolved feature map, and then cascade with the 26x26 feature map of the 16th layer, that is, a route layer in this process, A convolutional layer, an up-sampling layer, and finally a route layer for cascading; after cascading, three convolutions are performed, and the yolo layer outputs the prediction results on the 26x26 scale. The input of the convolutional neural network model is a video frame, and the output is a prediction result on two scales of 13x13 and 26x26. The prediction result is a three-dimensional tensor (tensor), and the three-dimensional tensor represents the object in the video. Boundary prediction box (that is, the vehicle position prediction box), the confidence level of the object and the object type, each boundary prediction box is composed of a vector, and the vector represents the center coordinates, width and height of the detected vehicle position prediction box.

Except for the route layer, other adjacent layers use the output of the previous layer as the input of the latter layer.

After the network design of the convolutional neural network model is completed, the network is written layer by layer, and then training and parameter adjustment are performed on the VOC (Visual Object Class, visual object class) data set until the convergence target is met, and the trained volume is obtained. A neural network model. The loss function used in the training process of the convolutional neural network model is as follows:

表示第j个边界预测框在网格i中。参数x、y、w、h、c、p中，不带上标参数的表示该参数的预测结果(例如参数x_i、y_i表示车辆位置预测框中心坐标的预测值)，带上标的表示的是该参数在ground truth(正确标记的数据)中对应的项(例如参数x_i'、y_i'表示车辆位置预测框中心坐标在ground truth中对应的项)。where x and y represent the center coordinates of the predicted vehicle position frame (ie, the vehicle position prediction frame), w and h respectively represent the width and height of the predicted vehicle position frame, and λ is the penalty coefficient. Penalty is performed when there is an object (obj) and when there is no object (noobj), S represents the division parameter of the image sent to the network, c is the confidence of the detected object, p(c) is the detected object belongs to the c class Confidence. In the formula, λ _coord and λ _noobj refer to the penalty coefficient, the former refers to the penalty coefficient when there is an object in the vehicle position prediction frame, and the latter refers to the penalty coefficient when there is no object in the vehicle position prediction frame. B represents the number of bounding prediction boxes.

Indicates that the jth bounding box is in grid i. In the parameters x, y, w, h, c, and p, the parameters without superscript represent the prediction result of the parameter (for example, the parameters x _i and y _i represent the predicted value of the center coordinates of the vehicle position prediction frame), and the superscript represents the prediction result of the parameter. is the corresponding item of the parameter in the ground truth (correctly labeled data) (for example, the parameters x _i ', y _i ' represent the corresponding item in the ground truth of the center coordinates of the vehicle position prediction box).

The trained convolutional neural network model is used to detect the center coordinates, width and height of the vehicle position frame (that is, the vehicle boundary prediction frame, also referred to as the vehicle position prediction frame in the present invention), and the confidence level of the vehicle (that is, the vehicle probability, It is used to compare with the confidence threshold in the algorithm) and the vehicle type (due to the one_hot (one-hot code) representation, if the detection object is a car, the item representing the vehicle is 1).

After the trained convolutional neural network model is generated, it can be directly loaded and called when detecting vehicles in the parking lot.

Step 103: Input the video frame into the trained convolutional neural network model, and output the vehicle position prediction frame information. The vehicle position prediction frame information includes the center coordinates, width and height of the vehicle position prediction frame.

The input of the convolutional neural network model is a video frame, and the output is a prediction result on two scales of 13x13 and 26x26. The prediction result is a three-dimensional tensor (tensor), and the three-dimensional tensor represents the object in the video. Boundary prediction box (that is, the vehicle position prediction box), the confidence level of the object and the object type, each boundary prediction box is composed of a vector, and the vector represents the center coordinates, width and height of the detected vehicle position prediction box. The current video frame collected by the camera is input into the trained convolutional neural network model, and the center coordinates, width and height of the vehicle position prediction frame can be directly output.

Step 104: Acquire parking space frame information. The parking space frame information includes the coordinates of the upper left corner and the lower right corner of the parking space frame.

At the beginning of the operation of the method and system of the present invention, the parking space needs to be extracted or marked to obtain the parking space information. Since the parking space exists in the background of the video frame, and the features are not obvious, it is difficult to extract through the neural network, so the present invention adopts the method of parameterized adjustment to obtain the parking space information. The detailed acquisition method is as follows:

设车位偏斜角度为α；设车位间的距离为δ。由于每个摄像头管理三到五个车位，所以还需设立一个参数β表示同一摄像头监测的车位的数目。所有的预置车位框看作一个整体，还需要一个参数设置这些预置车位框的一个初始位置，本发明只需要设定这个整体的一个角的位置即可，本发明实施例中设置的是车位框左上角的位置坐标，设该坐标点为p。Obtain the aspect ratio information of the parking space, set the width of the parking space as w and the height of the parking space as h, then the aspect ratio of the parking space

Let the deflection angle of the parking spaces be α; let the distance between the parking spaces be δ. Since each camera manages three to five parking spaces, a parameter β needs to be established to represent the number of parking spaces monitored by the same camera. All the preset parking space frames are regarded as a whole, and a parameter is needed to set an initial position of these preset parking space frames. The present invention only needs to set the position of one corner of the whole. The position coordinate of the upper left corner of the parking space frame, set the coordinate point as p.

First, the width and height of the parking space frame are initialized as w ₀ and h ₀ respectively. Since different parking lots may have different sizes of parking spaces, the aspect ratio ε is usually given by the parking lot administrator. In fact, through the setting of ε, only need It is sufficient to know either w ₀ and h ₀ . Assuming that only h ₀ is set, then w ₀ =ε*h ₀ , and the skew angle of the parking space is set to α ₀ (generally, the parking space angle is set to a right angle. If the deflection of the parking space angle is large, it can be adjusted appropriately by this parameter) , the initial value of the interval δ of the parking space frame is δ ₀ , the number of parking spaces is initialized as β ₀ , and the position coordinate of the upper left corner of the parking space frame is p ₀ (x ₀ , y ₀ ). Therefore, in the initialization stage, in the image of the camera, firstly, β ₀ is fixed to an initialized preset parking space frame with an aspect ratio of ε, a length of h ₀ (w ₀ can be obtained by inference according to ε), and a parking space frame interval of δ _0. When After the camera is installed, by adjusting p ₀ (x ₀ , y ₀ ), put all the parking space frames in the row where the parking spaces are located in the camera screen, and then adjust the ε, α, δ, h, β by adjusting the parking space frame. position, so that the preset parking space frame in the camera screen is adjusted to approximately the same parking line as the actual parking space. After all the parking space frames are adjusted to appropriate positions, the positions of different parking spaces and the corresponding parking space frame information can be obtained. The parking space frame information includes the coordinates of the upper left corner and the lower right corner of the parking space frame.

Step 105: Determine the current parking space state according to the vehicle position prediction frame information and the parking space frame position information.

In the present invention, the position of the vehicle is represented by Cartesian coordinates, and is represented by the upper left corner coordinate point and the lower right corner coordinate point of the vehicle position prediction frame predicted by the trained convolutional neural network model. The representation of the parking space is also established in the Cartesian coordinate system. The coordinates of the four corners are extracted and recorded. When the system starts, the corresponding parking space in each camera is acquired, and the corresponding The information of the parking space is saved and recorded in the memory at the same time, so that the information of the parking space can be obtained in time when the real-time analysis of the returned picture is performed. The x-axis of the Cartesian coordinate system is the width (w) direction of the video frame, the y-axis is the height (i.e. h) direction of the video frame, and the positive direction of the y-axis in the Cartesian coordinate system established in the present invention should be noted. down instead of the conventional up.

In the present invention, (x, y, w, h) is used to represent the prediction result predict_car of the trained convolutional neural network model, x and y represent the coordinates of the upper left corner of the vehicle position prediction frame, and w and h represent the vehicle respectively. The width and height of the position prediction frame, (obj_x_topleft, obj_y_topleft), (obj_x_bottomright, obj_y_bottomright) represent the upper left corner and lower right corner coordinates of the vehicle position prediction frame, respectively, (x_topleft, y_topleft), (x_bottomright, y_bottomright) respectively represent the parking space frame The coordinates of the upper left corner and the lower right corner, (x_toplo, y_toplo), (x_bottomro, y_bottomro) respectively represent the upper left corner coordinates and the lower right corner coordinates of the overlapping area of the vehicle position prediction frame and the parking space frame, and the ratio is n, judged The process of whether the parking space is empty is as follows:

Step 1: Calculate the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame according to the vehicle position prediction frame information.

The coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame need to be calculated according to the x, y, w, and h in the model prediction result. The actual calculation method is: obj_x_topleft=x–w/2; obj_y_topleft=y–h/2; obj_x_bottomright=obj_x_topleft+w/2; obj_y_bottomright=obj_y_topleft+h/2.

Step 2: Calculate the overlapping area position of the vehicle position prediction frame and the parking space frame according to the upper left corner coordinates and the lower right corner coordinates of the vehicle position prediction frame and the upper left corner coordinates and the lower right corner coordinates of the parking space frame; The overlapping area position includes coordinates of the upper left corner and the lower right corner of the overlapping area of the vehicle position prediction frame and the parking space frame.

Take the maximum value of the coordinates in the x direction of the upper left corner of the prediction frame and the parking space frame as x_toplo, and take the maximum value of the y direction in the upper left corner of the prediction frame and the parking space frame as y_toplo, so as to obtain the upper left corner of the overlapping area coordinates (x_toplo, y_toplo) ; Then take the minimum value of the coordinates in the x direction of the lower right corner of the prediction frame and the parking space frame as x_bottomro, and take the minimum value of the y direction of the lower right corner of the prediction frame and the parking space frame as y_bottomro, so as to obtain the coordinates of the lower right corner of the overlapping area (x_bottomro, y_bottomro).

Step 3: Calculate the area of the overlapping area according to the position of the overlapping area.

The calculation formula of the overlap area overlap_area is overlap_area=abs((x_toplo-x_bottomro)*(y_toplo-y_bottomro)). The abs in the formula means to take the absolute value.

Step 4: Calculate the area of the parking space frame according to the position of the parking space frame.

The calculation formula of the parking space frame area parking_area is parking_area=abs((x_topleft−x_bottomright)*(y_topleft−y_bottomright)).

Step 5: Calculate the ratio η of the area of the overlapping area to the area of the parking space frame as the confidence level of the vehicle.

The calculation formula of confidence η is η=overlap_area/parking_area.

Step 6: Determine whether the parking space is empty according to the threshold and the ratio η.

It is judged whether the confidence of the vehicle is greater than or equal to the confidence threshold, and if so, it is determined that the current parking space state is an occupied state; if not, it is determined that the current parking space state is an empty state.

Initially, two identical two-dimensional arrays are set up to represent the parking space status. The two-dimensional array is actually a matrix used to represent the parking space status. The matrices are named old and new respectively. Different rows in the matrix represent different Camera, different columns represent different parking spaces corresponding to the camera, the old array is used to record the parking space status of the previous state, and the new is used to record the parking space status of the current state. The parking space state includes two states, namely empty and not empty (or occupied) states, which are represented by 0 and 1 respectively. The change of the data in the array new is judged according to the model detection result and the parking space information.

That is, the program flow of step 105 is shown in Table 2 below:

Table 2 The flow chart of judging whether the parking space is empty

Step 106: Determine the parking fee according to the current parking space state.

According to the state of the number of the same position on the old and new matrices, the change of the parking space state is judged, that is, the change of the parking space state is represented by the different data changes from old to new in the same position. For example: when it changes from 0 to 1, it means that the parking space has changed from an empty state to an occupied state, and the starting parking time of the vehicle is recorded at this time; from 1 to 0, it means that the parking space has changed from being occupied to an empty state, and then the end parking time is recorded. ; Thereby, the toll amount is calculated according to the starting parking time and the ending parking time (when it changes from 1 to 0).

That is, the step 106 determines the parking fee according to the current parking space status, which specifically includes:

Obtain the starting parking time when the current parking space state changes from an empty state to an occupied state;

Obtain the ending parking time when the current parking space state changes from the occupied state to the empty state;

Calculate the parking time of the vehicle according to the start parking time and the end parking time;

The parking fee is determined according to the parking time.

Aiming at the requirements of intelligent parking lot management, the method of the invention designs a convolutional neural network model to locate the vehicle based on the Darknet framework, and realizes the real-time monitoring of the parking status on the parking space by calculating the overlap degree of the predicted vehicle position and the parking space. Statistics of parking fees are carried out according to the status of the parking spaces. By applying the deep learning method to the management of parking resources, the present invention realizes intelligent and automatic monitoring of parking status and parking fees, which not only alleviates urban traffic congestion, effectively regulates the use of parking resources, but also enables the management of parking resources. More convenient and intelligent, liberating manpower.

Based on the intelligent parking management method provided by the present invention, the present invention also provides an intelligent parking management system based on deep learning. FIG. 5 is a structural diagram of a deep learning-based intelligent parking management system provided by the present invention. Referring to FIG. 5 , the system includes:

A video acquisition module 501, configured to acquire video frames collected by a camera in a parking lot;

A model loading module 502, used for loading the trained convolutional neural network model;

The vehicle position prediction module 503 is used to input the video frame into the trained convolutional neural network model, and output the vehicle position prediction frame information; the vehicle position prediction frame information includes the center coordinates of the vehicle position prediction frame and the width, high;

a parking space frame obtaining module 504, configured to obtain parking space frame information; the parking space frame information includes the coordinates of the upper left corner and the lower right corner of the parking space frame;

The current parking space state judgment module 505 is configured to determine the current parking space state according to the vehicle position prediction frame information and the parking space frame position information;

The parking fee determination module 506 is configured to determine the parking fee according to the current parking space state.

The system also includes a model establishment module, and the model establishment module specifically includes:

A model establishment unit for establishing a convolutional neural network model based on the Darknet framework and the YOLO algorithm; the convolutional neural network model includes a feature extractor and a detector;

The model training unit is used to train and adjust parameters of the convolutional neural network model by using the VOC data set to generate a trained convolutional neural network model.

Wherein, the current parking space status judgment module 505 specifically includes:

a vehicle position calculation unit, configured to calculate the upper left corner coordinate and the lower right corner coordinate of the vehicle position prediction frame according to the vehicle position prediction frame information;

an overlapping area position calculation unit, configured to calculate the overlap between the vehicle position prediction frame and the parking space frame according to the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame and the coordinates of the upper left corner and the lower right corner of the parking space frame area position; the overlapping area position includes the coordinates of the upper left corner and the lower right corner of the overlapping area of the vehicle position prediction frame and the parking space frame;

an overlapping area area calculation unit, configured to calculate the overlapping area area according to the overlapping area position;

a parking space frame area calculation unit, configured to calculate the parking space frame area according to the position of the parking space frame;

a confidence degree calculation unit, configured to calculate the ratio of the area of the overlapping area to the area of the parking space frame as the confidence degree of the vehicle;

a confidence level judgment unit, configured to judge whether the confidence level of the vehicle is greater than or equal to a confidence level threshold, and obtain a first judgment result;

an occupied state judgment unit, configured to determine that the current parking space state is an occupied state if the first judgment result is that the confidence level of the vehicle is greater than or equal to a confidence level threshold;

An empty state judging unit, configured to determine that the current parking space state is an empty state if the confidence level of the vehicle is less than the confidence level threshold.

The parking fee determination module 506 specifically includes:

a parking start time recording unit, used to obtain the start parking time when the current parking space state changes from an empty state to an occupied state;

an end parking time recording unit, used to obtain the end parking time when the current parking space state changes from an occupied state to an empty state;

a parking time calculation unit, configured to calculate the parking time of the vehicle according to the start parking time and the end parking time;

A parking fee calculation unit, configured to determine the parking fee according to the parking time.

The system of the invention uses the research results of the frontier technology of computer vision to design a convolutional neural network for dynamic identification, real-time positioning and detection of vehicles, and real-time tracking of vehicles entering and leaving the parking lot and timing charging; The camera monitors, and detects the video frames returned by each camera in real time, calculates the ratio of the overlap between the vehicle and the parking space and the parking space, and conducts real-time statistics on the parking space status, and finally displays the parking space status information and hourly charging information separately. in the screen where the corresponding camera is located. Through the unified integrated software system, the parking status of the parking lot is displayed in real time, so as to realize the intelligent management of the parking lot.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present invention; meanwhile, for those skilled in the art, according to the present invention There will be changes in the specific implementation and application scope. In conclusion, the contents of this specification should not be construed as limiting the present invention.

Claims (4)

1. a kind of intelligent parking management method based on deep learning, is characterized in that, described method comprises: Get the video frame captured by the parking lot camera; Load the trained convolutional neural network model; Inputting the video frame into the trained convolutional neural network model, and outputting vehicle position prediction frame information; the vehicle position prediction frame information includes the center coordinates, width and height of the vehicle position prediction frame; Obtain parking space frame information; the parking space frame information includes the coordinates of the upper left corner and the lower right corner of the parking space frame; the method of parameterized adjustment is used to obtain the information of the parking space frame, and the detailed acquisition method is as follows: Obtain the aspect ratio information of the parking space, set the width of the parking space as w and the height of the parking space as h, then the aspect ratio of the parking space

Let the parking space skew angle be α; let the distance between parking spaces be δ; since each camera manages three to five parking spaces, a parameter β is set to represent the number of parking spaces monitored by the same camera; all the preset parking space frames are regarded as one As a whole, a parameter is also needed to set an initial position of these preset parking space frames. It is only necessary to set the position of one corner of the whole, and set the position coordinates of the upper left corner of the parking space frame to p(x, y); First, the width and height of the parking space frame are initialized as w ₀ and h ₀ respectively. Due to the different sizes of parking spaces in different parking lots, the aspect ratio ε is usually given by the parking lot administrator; in fact, through the setting of ε, only need to know Any one of w ₀ and h ₀ is sufficient; if only h ₀ is set, then w ₀ =ε*h ₀ ; the skew angle of the parking space is set to α ₀ , the initial value of the interval δ of the parking space frame is δ ₀ , the The number is initialized to β ₀ , and the position coordinates of the upper left corner of the parking space frame are p ₀ (x ₀ , y ₀ ); thus, in the initialization stage, the aspect ratio of β ₀ is first fixed as ε, the length is h ₀ , The initialized preset parking space frame with the parking space frame interval is δ _0. After the camera is installed, by adjusting p ₀ (x ₀ , y ₀ ), all the parking space frames are placed in the row where the parking space is located in the camera screen, and then pass through Adjust ε, α, δ, h, β to adjust the position of the parking space frame, so that the preset parking space frame in the camera screen is adjusted to be roughly the same as the parking line of the actual parking space. The adjustment range is that the preset parking space frame can represent the actual parking space. Therefore, the collected video frames can be detected more quickly; when all the parking space frames are adjusted to appropriate positions, the positions of different parking spaces and the corresponding parking space frame information can be obtained; Determining the current parking space state according to the vehicle position prediction frame information and the parking space frame position information; and determining the current parking space state according to the vehicle position prediction frame information and the parking space frame position information, specifically including: Calculate the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame according to the vehicle position prediction frame information; The position of the overlapping area between the vehicle position prediction frame and the parking space frame is calculated according to the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame, and the coordinates of the upper left corner and the lower right corner of the parking space frame; the overlapping area position Including the coordinates of the upper left corner and the lower right corner of the overlapping area of the vehicle position prediction frame and the parking space frame; Calculate the overlapping area area according to the overlapping area position; Calculate the area of the parking space frame according to the position of the parking space frame; Calculate the ratio of the area of the overlapping area to the area of the parking space frame as the confidence level of the vehicle; Determine whether the confidence of the vehicle is greater than or equal to a confidence threshold, and obtain a first judgment result; If the first judgment result is that the confidence of the vehicle is greater than or equal to the confidence threshold, determine that the current parking space state is an occupied state; If the confidence level of the vehicle is less than the confidence level threshold, it is determined that the current parking space state is an empty state. 2. The intelligent parking management method based on deep learning according to claim 1, is characterized in that, before described loading the trained convolutional neural network model, also comprises: A convolutional neural network model is established based on the Darknet framework and the YOLO algorithm; the convolutional neural network model includes a feature extractor and a detector; The convolutional neural network model is trained and adjusted by using the visual object class VOC data set to generate a trained convolutional neural network model. 3. An intelligent parking management system based on deep learning, wherein the system comprises: The video capture module is used to obtain the video frames captured by the parking lot camera; The model loading module is used to load the trained convolutional neural network model; The vehicle position prediction module is used to input the video frame into the trained convolutional neural network model, and output the vehicle position prediction frame information; the vehicle position prediction frame information includes the center coordinates of the vehicle position prediction frame and the width and height of the frame. ; The parking space frame acquisition module is used to obtain the parking space frame information; the parking space frame information includes the coordinates of the upper left corner and the lower right corner of the parking space frame; the parameterized adjustment method is used to obtain the parking space frame information, and the detailed acquisition method is as follows: Obtain the aspect ratio information of the parking space, set the width of the parking space as w and the height of the parking space as h, then the aspect ratio of the parking space

Let the parking space skew angle be α; let the distance between parking spaces be δ; since each camera manages three to five parking spaces, a parameter β is set to represent the number of parking spaces monitored by the same camera; all the preset parking space frames are regarded as one As a whole, a parameter is also needed to set an initial position of these preset parking space frames. It is only necessary to set the position of one corner of the whole, and set the position coordinates of the upper left corner of the parking space frame to p(x, y); First, the width and height of the parking space frame are initialized as w ₀ and h ₀ respectively. Due to the different sizes of parking spaces in different parking lots, the aspect ratio ε is usually given by the parking lot administrator; in fact, through the setting of ε, only need to know Any one of w ₀ and h ₀ is sufficient; if only h ₀ is set, then w ₀ =ε*h ₀ ; the skew angle of the parking space is set to α ₀ , the initial value of the interval δ of the parking space frame is δ ₀ , the The number is initialized to β ₀ , and the position coordinates of the upper left corner of the parking space frame are p ₀ (x ₀ , y ₀ ); thus, in the initialization stage, the aspect ratio of β ₀ is first fixed as ε, the length is h ₀ , The initialized preset parking space frame with the parking space frame interval is δ _0. After the camera is installed, by adjusting p ₀ (x ₀ , y ₀ ), all the parking space frames are placed in the row where the parking space is located in the camera screen, and then pass through Adjust ε, α, δ, h, β to adjust the position of the parking space frame, so that the preset parking space frame in the camera screen is adjusted to be roughly the same as the parking line of the actual parking space. The adjustment range is that the preset parking space frame can represent the actual parking space. Therefore, the collected video frames can be detected more quickly; when all the parking space frames are adjusted to appropriate positions, the positions of different parking spaces and the corresponding parking space frame information can be obtained; The current parking space state judgment module is configured to determine the current parking space state according to the vehicle position prediction frame information and the parking space frame position information; The current parking space status judgment module specifically includes: a vehicle position calculation unit, configured to calculate the upper left corner coordinate and the lower right corner coordinate of the vehicle position prediction frame according to the vehicle position prediction frame information; an overlapping area position calculation unit, configured to calculate the overlap between the vehicle position prediction frame and the parking space frame according to the coordinates of the upper left corner and the lower right corner of the vehicle position prediction frame and the coordinates of the upper left corner and the lower right corner of the parking space frame area position; the overlapping area position includes the coordinates of the upper left corner and the lower right corner of the overlapping area of the vehicle position prediction frame and the parking space frame; an overlapping area area calculation unit, configured to calculate the overlapping area area according to the overlapping area position; a parking space frame area calculation unit, configured to calculate the parking space frame area according to the position of the parking space frame; a confidence degree calculation unit, configured to calculate the ratio of the area of the overlapping area to the area of the parking space frame as the confidence degree of the vehicle; a confidence level judgment unit, configured to judge whether the confidence level of the vehicle is greater than or equal to a confidence level threshold, and obtain a first judgment result; an occupied state judgment unit, configured to determine that the current parking space state is an occupied state if the first judgment result is that the confidence level of the vehicle is greater than or equal to a confidence level threshold; An empty state judging unit, configured to determine that the current parking space state is an empty state if the confidence level of the vehicle is less than the confidence level threshold. 4. The intelligent parking management system based on deep learning according to claim 3, wherein the system further comprises a model building module, and the model building module specifically comprises: A model establishment unit for establishing a convolutional neural network model based on the Darknet framework and the YOLO algorithm; the convolutional neural network model includes a feature extractor and a detector; The model training unit is used to train and adjust parameters of the convolutional neural network model by using the VOC data set to generate a trained convolutional neural network model.

Images