CN116012817A - Real-time panoramic parking space detection method and device based on double-network deep learning - Google Patents
Real-time panoramic parking space detection method and device based on double-network deep learning Download PDFInfo
- Publication number
- CN116012817A CN116012817A CN202310104490.9A CN202310104490A CN116012817A CN 116012817 A CN116012817 A CN 116012817A CN 202310104490 A CN202310104490 A CN 202310104490A CN 116012817 A CN116012817 A CN 116012817A
- Authority
- CN
- China
- Prior art keywords
- parking space
- deep learning
- parking
- result set
- candidate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention provides a real-time panoramic parking space detection method based on double-network deep learning, which has good detection effect on a diagonal parking space and supports parking space detection under the condition of no parking space line and insufficient parking space line, and comprises the steps of calling a trained deep learning parking space instance segmentation model, detecting a real-time panoramic video frame and obtaining a segmented parking space line result; extracting characteristic points for fitting to obtain a parking space line; logically judging and combining the obtained parking space lines according to the constraint of the conventional parking space to form a parking space, and storing a candidate parking space result set P1 obtained by deeply learning a parking space instance segmentation model; calling a trained deep learning parking space corner detection model to detect and obtain a parking space corner detection result; combining the accurate angle values obtained by the parking space segmentation, and calculating to obtain a candidate parking space result set P2; carrying out comprehensive logic judgment and screening out a detection parking space set with the best confidence coefficient; and tracking the parking spaces obtained by detecting the historical frames by combining dead reckoning to carry out position compensation.
Description
Technical Field
The invention relates to the technical field of auxiliary parking, in particular to a real-time panoramic parking space detection method, device and storage medium based on double-network deep learning.
Background
With the rapid development of intellectualization and deep learning, automatic parking, remote control parking and memory parking are increasingly widely used. The advantage of deep learning is more remarkable, so that more researches are obtained on the basis of the parking space detection of the deep learning vision, and particularly, the memory parking has higher recognition rate requirements on the parking space detection, so that a more perfect map can be constructed, and the parking space information can be stored.
The current common panoramic parking space detection network based on deep learning comprises a deep learning network for detecting parking space corner points and a deep learning network for dividing parking space lines;
only the deep learning network for detecting the parking space corner points has better detection effect on right-angle parking spaces at present, is not ideal for the detection effect on non-right-angle inclined train positions, and particularly, when the included angle of the inclined train positions is returned, the obtained included angle and the actual scene have access, and finally the effect after complete parking can be influenced, so that the vehicle cannot park in the middle. Some scenes have no parking space corner information or the parking space corner is severely worn away, which is challenging for parking space corner detection;
the method has the advantages that only the deep learning network for dividing the parking space lines can well solve the problem of inaccurate positioning of the inclined train position included angles, but the dividing method is obviously more than the detection of the parking space corner points for the parking space lines or the masonry of the combined parking space, and the detection effect is not ideal because of insufficient information of the parking space lines in certain special cases.
Disclosure of Invention
Aiming at the problems, the invention provides a real-time panoramic parking space detection method, a device and a storage medium based on double-network deep learning, which support detection of various parking space types, have good detection effect on the inclined parking spaces and support parking space detection under the condition of no parking space line and incomplete parking space line.
The technical scheme is as follows: the real-time panoramic parking space detection method based on double-network deep learning is characterized by comprising the following steps of:
acquiring video data acquired by cameras in front, back, left and right directions of a panoramic system vehicle in real time, and synthesizing to obtain a real-time panoramic video frame;
calling a trained deep learning parking space example segmentation model, detecting a real-time panoramic video frame, and obtaining a segmented parking space line result;
extracting characteristic points from the parking space line results of the transverse traversal and the longitudinal traversal segmentation respectively; obtaining parking spot feature points at any angle;
fitting the obtained characteristic points according to the transverse direction and the longitudinal direction to obtain a parking space line with any angle;
logically judging and combining the obtained parking space lines according to the constraint of conventional parking spaces to form parking spaces, wherein the conventional parking spaces comprise parallel parking spaces, vertical parking spaces, inclined right-angle parking spaces and inclined non-right-angle parking spaces, and storing a candidate parking space result set P1 obtained through a deep learning parking space instance segmentation model;
calling a trained deep learning parking space corner detection model, and detecting a real-time panoramic video frame to obtain a parking space corner detection result;
matching the detected parking space corner points, and calculating to obtain a candidate parking space result set P2 by combining the accurate angle values obtained by the parking space segmentation;
carrying out comprehensive logic judgment on the candidate parking space result set P1 and the candidate parking space result set P2 obtained by detection, and finally screening out the detection parking space set with the best confidence coefficient;
and tracking the parking spaces obtained by detecting the historical frames by combining dead reckoning to carry out position compensation.
Further, a deep learning parking space corner detection model is constructed based on a convolutional neural network, the deep learning parking space corner detection model is used for learning features of parking space corners, a target frame containing the detected parking space corners and the type and the confidence of the target frame are output, and the target frame is represented by an upper left corner coordinate and a lower right corner coordinate of the target frame in an original image.
Further, when a training set of the deep learning parking space corner detection model is constructed, the parking space corner points in the synthesized panoramic image are marked with target frames and stored in addition, the storage path of the original image is put into an original image list, the storage path of the panoramic image marked with the parking space corner points is put into a marked image list, and the deep learning parking space corner detection model is called to load the original image list and the corresponding marked image list for iterative training.
Further, the labels of the parking space corner points comprise vertical T-shaped, vertical L-shaped, inclined T-shaped, inclined L-shaped, cross-shaped, U-shaped, linear type, disabled parking spaces, forbidden parking spaces P and forbidden parking spaces X.
Further, the comprehensive logic judgment is performed on the candidate parking space result set P1 and the candidate parking space result set P2 obtained by detection, and finally, the detected parking space set with the best confidence is selected, and the method is specifically implemented as follows:
if the candidate parking space result set P1 and the candidate parking space result set P2 are overlapped, finding all four parking space lines of the parking space corresponding to the parking space according to the parking space line serial numbers stored in the candidate parking space result set P1, judging whether one of the four parking space lines passes through two parking space corner points of the parking space in the candidate parking space result set P2, if so, considering that the parking space position in the candidate parking space result set P2 is higher, calculating the other two parking space corner points of the parking space in the candidate parking space result set P2 according to the included angle of the parking space in the candidate parking space result set P1, finally outputting the parking space in the candidate parking space result set P2, setting the parking space in the candidate parking space result set P2 to be credible, setting the parking space in the candidate parking space result set P1 to be unreliable, and if not, directly outputting the parking space in the candidate parking space result set P1 to be credible;
if the candidate parking space result set P1 and the candidate parking space result set P2 are not overlapped, calculating the other two parking space corner points of the parking space in the candidate parking space result set P2 according to the included angle of the parking spaces on the same side of the historical frame, and simultaneously setting the parking spaces in the candidate parking space result set P1 and the candidate parking space result set P2 as trusted parking spaces and outputting the trusted parking spaces.
Further, when the coordinate difference value of the two parking space corner points of the candidate parking space result set P1 and the candidate parking space result set P2 is within the range of 50cm, the candidate parking space result set P1 and the candidate parking space result set P2 are considered to have parking space overlapping.
Further, when feature points are extracted according to the results of traversing and dividing the parking space lines in the transverse direction and the longitudinal direction, the feature points are slidingly extracted according to the local gray gradient values above and below the parking space lines, the transverse traversing range is [0 degrees, 45 degrees ] and [135 degrees, 180 degrees ], the longitudinal traversing range is [45 degrees, 135 degrees ], and the traversing angle range is an included angle with the horizontal transverse line.
Further, a deep learning parking space example segmentation model is built based on a convolutional neural network, the deep learning parking space example segmentation model comprises a Bias layer, a combination module of a Convolume layer, a BatchNorm layer and a Relu layer, a pulling layer, a Deconvolution layer and an Eltwise layer, the deep learning parking space example segmentation model learns each pixel of an input sample through Convolution downsampling, then a panoramic segmentation result diagram with the same size as that of an original image is input through Deconvolution output, a training set is arranged to train the deep learning parking space example segmentation model until the model converges to obtain a trained deep learning parking space example segmentation model.
Further, when the training set is constructed, the parking space lines in the synthesized panoramic image are marked and stored, the storage path of the original image is put into an original image list, the storage path of the panoramic image marked with the parking space lines is put into a segmentation label image list, and a deep learning parking space instance segmentation model is called to load the original image list and a corresponding segmentation label image list for iterative training.
Further, when the vehicle is logically judged and combined into the vehicle according to the constraint of the conventional vehicle, the constraint conditions of the parallel vehicle are simultaneously satisfied:
c1.1: the parking space included angle formed by the parking space lines is 90 degrees;
c1.2: the distance between two parking space corner points close to the vehicle is more than 450cm and less than 650cm;
c1.3: the connecting line of the two parking space corner points close to the vehicle is parallel to the vehicle;
the constraint conditions for judging the vertical parking spaces are simultaneously satisfied:
c2.1: the included angle of the parking space formed by the parking space lines is 90 degrees;
c2.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c2.3: the connecting line of two corner points close to the vehicle is parallel to the vehicle;
the constraint conditions of the inclined right-angle parking spaces are judged to be simultaneously satisfied:
c3.1: the parking space included angle formed by the parking space lines is 90 degrees;
c3.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c3.3: the included angle between the connecting line of the two parking space angular points close to the vehicle and the vehicle is kept at [30 degrees, 60 degrees ].
The constraint condition for judging the inclined non-right-angle parking spaces is simultaneously satisfied:
c4.1: when the parking space line forms a parking space included angle [30 degrees, 60 degrees ];
c4.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c4.3: the line of the two corner points near the vehicle is kept parallel to the vehicle.
Further, when the included angle between the connecting line of the two parking space corner points close to the vehicle and one of the coordinate axes taking the center of the vehicle as a coordinate system is smaller than 5 degrees, the connecting line of the two corner points of the vehicle is considered to be parallel to the vehicle.
Further, tracking the parking space detected by the historical frame by combining with the dead reckoning to perform position compensation specifically comprises the following steps:
the method comprises the steps that a vehicle signal is input through an external interface, the vehicle signal comprises a gear signal, a left rear wheel speed pulse LP, a right rear wheel speed pulse RP and a time stamp, a compensation value is obtained by multiplying a difference value of front and rear frame time stamps by the wheel speed pulse, and whether the position coordinate of a parking space at the moment is added with the compensation value or subtracted with the compensation value is determined according to whether the gear signal is a forward gear or a reverse gear, so that the position of a current frame of the parking space is synchronized.
A computer apparatus, comprising: comprises a processor, a memory and a program;
the program is stored in the memory, and the processor calls the program stored in the memory to execute the real-time panoramic parking space detection method based on the double-network deep learning.
A computer-readable storage medium, characterized by: the computer readable storage medium is used for storing a program for executing the real-time panoramic parking space detection method based on double-network deep learning.
The method combines the detection of the corner points of the deep learning parking space, the example segmentation of the line of the deep learning parking space and the traditional post-processing method for extracting the parking space, can simultaneously detect parallel parking spaces, vertical parking spaces, inclined non-right-angle parking spaces and inclined right-angle parking spaces, and supports the detection of the parking spaces under the condition of no line of the parking spaces and incomplete line of the parking spaces;
the invention realizes the fusion of the two deep learning detection methods, makes up the advantages and disadvantages of each method, has higher detection confidence of the parking space corner points, combines and matches the parking space by using the parking space corner points, and calculates the parking space lines with split precise angles of the parking space matched and combined by the parking space corner points; the position of the split parking space line is higher, and the split parking space line is used for matching the combined parking space;
according to the method, the position information of the split parking spaces can be quickly and accurately positioned through the deep learning parking space example split model, the position information of the parking space corner points can be quickly positioned through the deep learning parking space corner point detection model, and if only one network model cannot detect the parking spaces, the detection of the other network model on the parking spaces is not affected, so that the detection and recognition rate of the whole parking spaces is greatly improved.
Drawings
Fig. 1 is a schematic diagram of steps of a real-time panoramic parking space detection method based on deep learning of dual networks in an embodiment;
fig. 2 is a schematic diagram of a parking space corner tag;
fig. 3 is an internal structural view of the computer device in one embodiment.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail below. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, based on the examples herein, which are within the scope of the protection sought by those of ordinary skill in the art without undue effort, are intended to be encompassed by the present application.
As shown in the background art, the existing deep learning parking space detection method has defects, the deep learning network for detecting the parking space corner points has an unsatisfactory detection effect on the non-right-angle inclined train position, the deep learning network for dividing the parking space lines is obviously more information than the parking space corner point detection for combining the parking space lines or masonry, and the parking space line information is insufficient and the detection effect is unsatisfactory for some special cases.
The invention provides a real-time panoramic parking space detection method based on double-network deep learning, which is shown in fig. 1 and comprises the following steps:
step 1: acquiring video data acquired by cameras in front, back, left and right directions of a panoramic system vehicle in real time, and synthesizing to obtain a real-time panoramic video frame;
step 2: calling a trained deep learning parking space example segmentation model, detecting a real-time panoramic video frame, and obtaining a segmented parking space line result;
step 3: extracting characteristic points from the parking space line results of the transverse traversal and the longitudinal traversal segmentation respectively; obtaining parking spot feature points at any angle;
step 4: fitting the obtained characteristic points according to the transverse direction and the longitudinal direction to obtain a parking space line with any angle;
step 5: logically judging and combining the obtained parking space lines according to the constraint of the conventional parking spaces to form parking spaces, wherein the conventional parking spaces comprise parallel parking spaces, vertical parking spaces, inclined right-angle parking spaces and inclined non-right-angle parking spaces, and storing a candidate parking space result set P1 obtained through a deep learning parking space instance segmentation model;
step 6: calling a trained deep learning parking space corner detection model, and detecting a real-time panoramic video frame to obtain a parking space corner detection result;
step 7: matching the detected parking space corner points, and calculating to obtain a candidate parking space result set P2 by combining the accurate angle values obtained by the parking space segmentation;
step 8: carrying out comprehensive logic judgment on the candidate parking space result set P1 and the candidate parking space result set P2 obtained by detection, and finally screening out the detection parking space set with the best confidence coefficient;
step 9: and tracking the parking spaces obtained by detecting the historical frames by combining dead reckoning to carry out position compensation.
In step 1, a fisheye camera video in four directions of front, rear, left and right of a vehicle is collected in real time, and a panoramic system is called to synthesize images collected by the cameras, so that a real-time panoramic video frame is obtained through synthesis;
in step 2, a deep learning parking space example segmentation model is built based on a convolutional neural network, the deep learning parking space example segmentation model comprises a Bias layer, a combination module of a Convolving layer, a BatchNorm layer and a Relu layer, a pulling layer, a Deconvolution layer and an Eltwise layer, the deep learning parking space example segmentation model learns each pixel of an input sample through Convolution downsampling, then a panoramic segmentation result diagram with the same size as an original image input is output through Deconvolution, the deep learning parking space example segmentation model classifies each pixel of the input panoramic car bitmap, 0 represents a non-parking space line, namely a background, and 1 represents a parking space line, namely a segmented target.
Setting a training set to train a deep learning parking space example segmentation model, marking and saving parking space lines in a synthesized panoramic image when the training set is constructed, putting a storage path of an original image into an original image list image txt, storing the storage path of each original image, putting the storage path of the panoramic image marked with the parking space lines into a segmentation label image list label. Txt, calling the deep learning parking space example segmentation model to load the original image list and a corresponding segmentation label image list for iterative training until the model converges to obtain the trained deep learning parking space example segmentation model.
In step 3, when feature points are extracted according to the results of the parking space lines which are traversed and segmented in the transverse direction and the longitudinal direction at the same time, the feature points are slidingly extracted according to local gray gradient values above and below the parking space lines, wherein the transverse traversing range is [0 degrees, 45 degrees ] and [135 degrees, 180 degrees ], the longitudinal traversing range is [45 degrees, 135 degrees ], and the traversing angle range is an included angle with the horizontal transverse line, so that the features of the parking space lines at any angle can be obtained, the detection of the parking spaces at any angle is realized, and the parking space detection device not only comprises right-angle parking spaces but also comprises non-right-angle parking spaces;
in the step 4, fitting the obtained characteristic points according to the transverse direction and the longitudinal direction to obtain a parking space line with any angle to form a characteristic straight line segment; judging whether the characteristic straight line segment belongs to a straight line or not can be judged according to the slope of the straight line; fitting the characteristic straight line segment belonging to a straight line into a straight line, and taking the obtained fitted straight line as a parking space line;
in step 5, the obtained parking space lines are logically judged and combined into parking spaces according to the constraint of conventional parking spaces, wherein the conventional parking spaces comprise parallel parking spaces, vertical parking spaces, inclined right-angle parking spaces and inclined non-right-angle parking spaces; when the combination parking space is logically judged according to the constraint of the conventional parking space, the constraint conditions of the parallel parking spaces are simultaneously satisfied:
c1.1: the parking space included angle formed by the parking space lines is 90 degrees;
c1.2: the distance between two parking space corner points close to the vehicle is more than 450cm and less than 650cm;
c1.3: the connecting line of the two parking space corner points close to the vehicle is parallel to the vehicle, and when the included angle between the connecting line of the two parking space corner points close to the vehicle and one coordinate axis taking the center of the vehicle as a coordinate system is smaller than 5 degrees, the connecting line of the two parking space corner points of the vehicle is considered to be parallel to the vehicle;
the constraint conditions for judging the vertical parking spaces are simultaneously satisfied:
c2.1: the included angle of the parking space formed by the parking space lines is 90 degrees;
c2.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c2.3: the connecting line of two corner points close to the vehicle is parallel to the vehicle, and when the angle between the connecting line of two corner points close to the vehicle and one coordinate axis taking the center of the vehicle as a coordinate system is smaller than 5 degrees, the connecting line of two corner points of the vehicle is considered to be parallel to the vehicle;
the constraint conditions of the inclined right-angle parking spaces are judged to be simultaneously satisfied:
c3.1: the parking space included angle formed by the parking space lines is 90 degrees;
c3.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c3.3: the included angle between the connecting line of the two parking space angular points close to the vehicle and the vehicle is kept at [30 degrees, 60 degrees ].
The constraint condition for judging the inclined non-right-angle parking spaces is simultaneously satisfied:
c4.1: when the parking space line forms a parking space included angle [30 degrees, 60 degrees ];
c4.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c4.3: and when the included angle between the two corner connecting lines of the parking space near the vehicle and one of the coordinate axes taking the center of the vehicle as a coordinate system is smaller than 5 degrees, the two corner connecting lines of the vehicle are considered to be parallel to the vehicle.
And judging and combining the vehicle positions through the constraint conditions respectively, and storing a candidate vehicle position result set P1 obtained through the deep learning vehicle position example segmentation model.
Besides the deep learning parking space example segmentation model in the step 2, the deep learning parking space corner detection model is adopted in the embodiment at the same time, so that the parking space corner position information is accurately obtained.
In step 6 of the embodiment, a deep learning parking space corner detection model is built based on a convolutional neural network, the deep learning parking space corner detection model comprises a convolutional layer, a BN layer, an activation function ReLU and a Pooling layer, the deep learning parking space corner detection model is built based on an SSD model, data conversion is carried out on the deep learning parking space corner detection model, floating points are calculated according to integer 8 bits, and therefore calculation efficiency and speed are improved, and a real-time effect is achieved; the deep learning parking space corner detection model is used for learning features of parking space corners, outputting a target frame containing detected parking space corners and types and confidence coefficients of the target frame, wherein the target frame is represented by an upper left corner coordinate and a lower right corner coordinate of the target frame in an original image, the confidence coefficients of the detected parking space corner target frame can be output, and the confidence coefficients output by the deep learning parking space corner detection model are the confidence coefficients of the detected parking space corner target frame.
When a training set of the deep learning parking space corner detection model is constructed, the parking space corner in the synthesized panoramic image is marked with a target frame and stored in addition, a storage path of an original image is put into an original image list, the storage path of the panoramic image marked with the parking space corner is put into a marked image list, and the deep learning parking space corner detection model is called to load the original image list and the corresponding marked image list for iterative training.
As shown in fig. 2, the types of parking space corner labels are classified into ten types in total in the embodiment, including a vertical T-type, a vertical L-type, a slanted T-type, a slanted L-type, a cross-type, a U-type, a straight-line type, a disabled person parking space, a forbidden parking space P, and a forbidden parking space X.
In step 7 of the embodiment, matching the detected parking space corner points, and calculating the other two parking space corner points of the parking space by combining the accurate angle values obtained by dividing the parking space to obtain a candidate parking space result set P2;
in step 8 of the embodiment, comprehensive logic judgment is performed on the candidate parking space result set P1 and the candidate parking space result set P2 obtained through detection, and finally, the detection parking space set with the best confidence coefficient is selected; the method is specifically implemented as follows:
for the parking stall that obtains in candidate parking stall result collection P1, this parking stall contains complete parking stall information, and the parking stall information that contains has: the parking space corner points, the parking space types, the parking space orientations and the parking space included angles are all calculated by the information of each segmented parking space line;
the parking space obtained in the candidate parking space result set P2 is obtained through corner detection, the included parking space information only includes two corner points of the parking space, and the parking space information is imperfect, because only two corner points of the parking space may be detected in the panoramic aerial view, at this time, the included angle of the parking space cannot be accurately judged, and therefore the other two corner points of the parking space cannot be calculated.
In the embodiment, when the coordinate difference between the two parking space corner points in the candidate parking space result set P1 and the candidate parking space result set P2 is within the range of 50cm, the candidate parking space result set P1 and the candidate parking space result set P2 are considered to have parking space overlapping.
If the candidate parking space result set P1 and the candidate parking space result set P2 are overlapped, finding all four parking space lines of the parking space corresponding to the parking space according to the parking space line serial numbers stored in the candidate parking space result set P1, judging whether one of the four parking space lines passes through two parking space corner points of the parking space in the candidate parking space result set P2, if so, considering that the parking space position in the candidate parking space result set P2 is higher, calculating the other two parking space corner points of the parking space in the candidate parking space result set P2 according to the included angle of the parking space in the candidate parking space result set P1, finally outputting the parking space in the candidate parking space result set P2, setting the parking space in the candidate parking space result set P2 to be credible, setting the parking space in the candidate parking space result set P1 to be unreliable, and if not, directly outputting the parking space in the candidate parking space result set P1 to be credible;
if the candidate parking space result set P1 and the candidate parking space result set P2 are not overlapped, calculating the other two parking space corner points of the parking space in the candidate parking space result set P2 according to the included angle of the parking spaces on the same side of the historical frame, and simultaneously setting the parking spaces in the candidate parking space result set P1 and the candidate parking space result set P2 as trusted parking spaces and outputting the trusted parking spaces.
And 8, simultaneously considering the parking space segmentation and the parking space corner detection to carry out comprehensive logic judgment on the parking spaces of all scenes, and obtaining an excellent detection result.
In step 8 in the embodiment, the parking space position degree includes a parking space line length L of the parking space, a parallelism P of opposite sides of the parking space, and a calculated parking space angular point and two nearest neighboring parking space line distance confidence degrees S, where the parking space position degree c=l=0.3+p+0.4+s+0.3 is calculated;
in step 9 of the embodiment, tracking the parking space detected by the history frame in combination with dead reckoning for position compensation specifically includes:
the vehicle signal is input through an external interface, the vehicle signal comprises a gear signal, a left rear wheel speed pulse LP, a right rear wheel speed pulse RP and a time stamp, the gear signal comprises a forward gear D and a backward gear R, a compensation value is obtained by multiplying the difference value of the front frame time stamp and the rear frame time stamp by the wheel speed pulse, and the position coordinate of a parking space at the moment is determined to be the compensation value or the compensation value is added or subtracted according to whether the gear signal is the forward gear or the backward gear, so that the position of the parking space in the current frame is synchronized, and the detected parking space is more stable and accurate through position compensation.
In the traditional method for detecting the parking space by non-deep learning, the method for extracting the feature points of the parking space is carried out according to the gradient difference value of the pixels of the image, namely the feature points are extracted according to the local difference value of the pixels of the input image, and the traditional method has influence on the fact that the illumination dependence is large, the illumination intensity is bright and dark, whether the ground reflects light, whether the pixel value of the parking space line and the pixel value of the nearby background are large, and the like; the method comprises the steps of dividing a parking space line by a deep learning parking space instance division model, extracting characteristic points on the divided parking space line, judging whether the characteristic points are the parking space line characteristic points according to a division result value, wherein the division value is 0 as the background, and the division value is 1 as the parking space line; then extracting characteristic points by using the results of traversing the split parking space lines transversely and longitudinally, fitting the obtained characteristic points according to the transverse direction and the longitudinal direction to obtain a parking space line at any angle, and obtaining a parking space through parking space constraint; compared with the traditional method, the method can better cope with the parking space detection on the ground under the conditions of no parking space line and insufficient parking space line, can support the detection of the parking space which is a masonry parking space on the ground and is partially covered by the weeds in the district or the detection and identification effect of the parking space under the shade of tree, and the method is used for processing the feature point extraction after deep learning, thereby accelerating the processing speed, reducing the running time and simultaneously detecting the parallel parking space, the vertical parking space, the inclined non-right-angle parking space and the inclined right-angle parking space; compared with other deep learning methods, the method has the advantages that one deep learning network is utilized to do two things, the parking space line is needed to be segmented, and the segmented parking space line is subjected to logistic regression positioning, so that the time required by the method is longer than that required by the method, the segmented deep learning parking space example segmentation model is only used for segmenting the parking space line, the segmented result is used for positioning the parking space by using a conventional algorithm, very strong hardware performance is not needed, the hardware dependence is low, and the calculation power requirement on a vehicle computer is low.
The method combines the detection of the corner points of the deep learning parking space, the example segmentation of the line of the deep learning parking space and the traditional post-processing method for extracting the parking space, can simultaneously detect parallel parking spaces, vertical parking spaces, inclined non-right-angle parking spaces and inclined right-angle parking spaces, and supports the detection of the parking spaces under the condition of no line of the parking spaces and incomplete line of the parking spaces;
the invention realizes the fusion of the two deep learning detection methods, makes up the advantages and disadvantages of each method, has higher detection confidence of the parking space corner points, combines and matches the parking space by using the parking space corner points, and calculates the parking space lines with the split precise angles of the parking spaces matched and combined by the parking space corner points. The position of the split parking space line is higher, and the split parking space line is used for matching the combined parking space;
according to the method, the position information of the split parking spaces can be quickly and accurately positioned through the deep learning parking space example split model, the position information of the parking space corner points can be quickly positioned through the deep learning parking space corner point detection model, and if only one network model cannot detect the parking spaces, the detection of the other network model on the parking spaces is not affected, so that the detection and recognition rate of the whole parking spaces is greatly improved.
At present, the real-time panoramic parking space detection method based on double-network deep learning can realize real-time parking space detection on TDA4 hardness, and meets the development requirements of users.
In an embodiment of the present invention, there is also provided a computer apparatus including: comprises a processor, a memory and a program;
the program is stored in the memory, and the processor calls the program stored in the memory to execute the real-time panoramic parking space detection method based on the double-network deep learning.
The computer device may be a terminal, and its internal structure may be as shown in fig. 3. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by the processor, implements a real-time panoramic parking detection method for deep learning based on dual networks. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer device, and can also be an external keyboard, a touch pad or a mouse and the like.
The Memory may be, but is not limited to, random access Memory (Random Access Memory; RAM; ROM; programmable Read-Only Memory; PROM; erasable ROM; erasable Programmable Read-Only Memory; EPROM; electrically erasable ROM; electric Erasable Programmable Read-Only Memory; EEPROM; etc.). The memory is used for storing a program, and the processor executes the program after receiving the execution instruction.
The processor may be an integrated circuit chip with signal processing capabilities. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), and the like. The processor may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be appreciated by those skilled in the art that the structure shown in fig. 3 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer apparatus to which the present application may be applied, and that a particular computer apparatus may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In an embodiment of the present invention, a computer readable storage medium is further provided, where the computer readable storage medium is used to store a program, and the program is used to execute the above-mentioned real-time panoramic parking space detection method based on deep learning of dual networks.
It will be appreciated by those skilled in the art that embodiments of the invention may be provided as a method, a computer device, or a computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations of methods, computer apparatus, or computer program products according to embodiments of the invention. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart.
The above describes the application of the dual-network-based deep learning real-time panoramic parking space detection method, the computer device and the computer readable storage medium in detail, and specific examples are applied to illustrate the principle and the implementation of the invention, and the above description of the examples is only used for helping to understand the method and the core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (13)
1. The real-time panoramic parking space detection method based on double-network deep learning is characterized by comprising the following steps of:
acquiring video data acquired by cameras in front, back, left and right directions of a panoramic system vehicle in real time, and synthesizing to obtain a real-time panoramic video frame;
calling a trained deep learning parking space example segmentation model, detecting a real-time panoramic video frame, and obtaining a segmented parking space line result;
extracting characteristic points from the parking space line results of the transverse traversal and the longitudinal traversal segmentation respectively; obtaining parking spot feature points at any angle;
fitting the obtained characteristic points according to the transverse direction and the longitudinal direction to obtain a parking space line with any angle;
logically judging and combining the obtained parking space lines according to the constraint of conventional parking spaces to form parking spaces, wherein the conventional parking spaces comprise parallel parking spaces, vertical parking spaces, inclined right-angle parking spaces and inclined non-right-angle parking spaces, and storing a candidate parking space result set P1 obtained through a deep learning parking space instance segmentation model;
calling a trained deep learning parking space corner detection model, and detecting a real-time panoramic video frame to obtain a parking space corner detection result;
matching the detected parking space corner points, and calculating to obtain a candidate parking space result set P2 by combining the accurate angle values obtained by the parking space segmentation;
and (3) carrying out comprehensive logic judgment on the candidate parking space result set P1 and the candidate parking space result set P2 which are obtained through detection, and finally screening out the detection parking space set with the best confidence.
2. The real-time panoramic parking space detection method based on double-network deep learning of claim 1, wherein the method comprises the following steps of: the method comprises the steps of constructing a deep learning parking space corner detection model based on a convolutional neural network, wherein the deep learning parking space corner detection model is used for learning features of parking space corners, outputting a target frame containing the detected parking space corners, and the type and the confidence of the target frame, wherein the target frame is represented by an upper left corner coordinate and a lower right corner coordinate of the target frame in an original image.
3. The real-time panoramic parking space detection method based on double-network deep learning of claim 2, wherein the method comprises the following steps of: when a training set of the deep learning parking space corner detection model is constructed, a target frame is marked and stored on the parking space corner in the synthesized panoramic image, a storage path of the original image is placed in an original image list, a storage path of the panoramic image marked with the parking space corner is placed in the marking image list, the deep learning parking space corner detection model is called to load the original image list and the corresponding marking image list for iterative training, and the labels of the parking space corner comprise a vertical T type, a vertical L type, an inclined T type, an inclined L type, a cross type, a U type, a linear type, a disabled parking space, a forbidden parking space P and a forbidden parking space X.
4. The method for detecting the real-time panoramic parking space based on the deep learning of the double networks according to claim 1, wherein the comprehensive logic judgment is carried out on the candidate parking space result set P1 and the candidate parking space result set P2 obtained by detection, and finally the detection parking space set with the best confidence is selected, and the method is specifically implemented as follows:
if the candidate parking space result set P1 and the candidate parking space result set P2 are overlapped, finding all four parking space lines of the parking space corresponding to the parking space according to the parking space line serial numbers stored in the candidate parking space result set P1, judging whether one of the four parking space lines passes through two parking space corner points of the parking space in the candidate parking space result set P2, if so, considering that the parking space position in the candidate parking space result set P2 is higher, calculating the other two parking space corner points of the parking space in the candidate parking space result set P2 according to the included angle of the parking space in the candidate parking space result set P1, finally outputting the parking space in the candidate parking space result set P2, setting the parking space in the candidate parking space result set P2 to be credible, setting the parking space in the candidate parking space result set P1 to be unreliable, and if not, directly outputting the parking space in the candidate parking space result set P1 to be credible;
if the candidate parking space result set P1 and the candidate parking space result set P2 are not overlapped, calculating the other two parking space corner points of the parking space in the candidate parking space result set P2 according to the included angle of the parking spaces on the same side of the historical frame, and simultaneously setting the parking spaces in the candidate parking space result set P1 and the candidate parking space result set P2 as trusted parking spaces and outputting the trusted parking spaces.
5. The real-time panoramic parking space detection method based on double-network deep learning of claim 4, wherein the method comprises the following steps of: when the coordinate difference value of two parking space corner points exists between the candidate parking space result set P1 and the candidate parking space result set P2 within the range of 50cm, the candidate parking space result set P1 and the candidate parking space result set P2 are considered to have parking space overlapping.
6. The real-time panoramic parking space detection method based on double-network deep learning of claim 1, wherein the method comprises the following steps of: when feature points are extracted according to the results of the parking space lines which are traversed and segmented in the transverse direction and the longitudinal direction, the feature points are extracted according to the local gray gradient values above and below the parking space lines in a sliding mode, the transverse traversing range is [0 degrees, 45 degrees ] and [135 degrees, 180 degrees ], the longitudinal traversing range is [45 degrees, 135 degrees ], and the traversing angle range is an included angle with the horizontal transverse line.
7. The dual-network-based deep learning real-time panoramic parking space detection method as claimed in claim 6, wherein the method comprises the following steps: the method comprises the steps of constructing a deep learning parking space instance segmentation model based on a convolutional neural network, wherein the deep learning parking space instance segmentation model comprises a Bias layer, a Convolving layer, a BatchNorm layer, a Relu layer combination module, a pulling layer, a Deconvolution layer and an Eltwise layer, learning each pixel of an input sample through convolutional downsampling, then outputting a full-scale panoramic segmentation result diagram with the same size as that of an original image through Deconvolution, setting a training set to train the deep learning parking space instance segmentation model until the model converges to obtain a trained deep learning parking space instance segmentation model.
8. The dual-network-based deep learning real-time panoramic parking space detection method as claimed in claim 7, wherein: when the training set is constructed, the parking space lines in the synthesized panoramic image are marked and stored in addition, the storage path of the original image is put into an original image list, the storage path of the panoramic image marked with the parking space lines is put into a segmentation tag image list, and a deep learning parking space instance segmentation model is called to load the original image list and a corresponding segmentation tag image list for iterative training.
9. The real-time panoramic parking space detection method based on double-network deep learning of claim 1, wherein the method comprises the following steps of: when the combination parking space is logically judged according to the constraint of the conventional parking space, the constraint conditions of the parallel parking spaces are simultaneously satisfied:
c1.1: the parking space included angle formed by the parking space lines is 90 degrees;
c1.2: the distance between two parking space corner points close to the vehicle is more than 450cm and less than 650cm;
c1.3: the connecting line of the two parking space corner points close to the vehicle is parallel to the vehicle;
the constraint conditions for judging the vertical parking spaces are simultaneously satisfied:
c2.1: the included angle of the parking space formed by the parking space lines is 90 degrees;
c2.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c2.3: the connecting line of two corner points close to the vehicle is parallel to the vehicle;
the constraint conditions of the inclined right-angle parking spaces are judged to be simultaneously satisfied:
c3.1: the parking space included angle formed by the parking space lines is 90 degrees;
c3.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c3.3: the included angle between the connecting line of two parking space angular points close to the vehicle and the vehicle is kept at [30 degrees, 60 degrees ];
the constraint condition for judging the inclined non-right-angle parking spaces is simultaneously satisfied:
c4.1: when the parking space line forms a parking space included angle [30 degrees, 60 degrees ];
c4.2: the distance between two parking space corner points close to the vehicle is more than 200cm and less than 350cm;
c4.3: the line of the two corner points near the vehicle is kept parallel to the vehicle.
10. The dual-network-based deep learning real-time panoramic parking space detection method as claimed in claim 9, wherein: when the included angle between the connecting line of the two parking space corner points close to the vehicle and one of the coordinate axes taking the center of the vehicle as a coordinate system is smaller than 5 degrees, the connecting line of the two corner points of the vehicle is considered to be parallel to the vehicle.
11. The real-time panoramic parking space detection method based on double-network deep learning of claim 1, wherein the method comprises the following steps of: the method for tracking the parking space obtained by detecting the historical frame by combining the dead reckoning to carry out position compensation specifically comprises the following steps:
the method comprises the steps that a vehicle signal is input through an external interface, the vehicle signal comprises a gear signal, a left rear wheel speed pulse LP, a right rear wheel speed pulse RP and a time stamp, a compensation value is obtained by multiplying a difference value of front and rear frame time stamps by the wheel speed pulse, and whether the position coordinate of a parking space at the moment is added with the compensation value or subtracted with the compensation value is determined according to whether the gear signal is a forward gear or a reverse gear, so that the position of a current frame of the parking space is synchronized.
12. A computer apparatus, comprising: comprises a processor, a memory and a program;
the program is stored in the memory, and the processor calls the program stored in the memory to execute the real-time panoramic parking space detection method based on the double-network deep learning.
13. A computer-readable storage medium, characterized by: the computer readable storage medium is used for storing a program for executing the real-time panoramic parking space detection method based on double-network deep learning.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310104490.9A CN116012817A (en) | 2023-02-13 | 2023-02-13 | Real-time panoramic parking space detection method and device based on double-network deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310104490.9A CN116012817A (en) | 2023-02-13 | 2023-02-13 | Real-time panoramic parking space detection method and device based on double-network deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116012817A true CN116012817A (en) | 2023-04-25 |
Family
ID=86037399
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310104490.9A Pending CN116012817A (en) | 2023-02-13 | 2023-02-13 | Real-time panoramic parking space detection method and device based on double-network deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116012817A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117068145A (en) * | 2023-10-17 | 2023-11-17 | 北京茵沃汽车科技有限公司 | Parking method, parking device, computing device and storage medium |
| CN117274952A (en) * | 2023-09-26 | 2023-12-22 | 镁佳(北京)科技有限公司 | Parking space detection method and device, computer equipment and storage medium |
-
2023
- 2023-02-13 CN CN202310104490.9A patent/CN116012817A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117274952A (en) * | 2023-09-26 | 2023-12-22 | 镁佳(北京)科技有限公司 | Parking space detection method and device, computer equipment and storage medium |
| CN117068145A (en) * | 2023-10-17 | 2023-11-17 | 北京茵沃汽车科技有限公司 | Parking method, parking device, computing device and storage medium |
| CN117068145B (en) * | 2023-10-17 | 2024-01-26 | 北京茵沃汽车科技有限公司 | Parking method, parking device, computing device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111460926B (en) | Video pedestrian detection method fusing multi-target tracking clues | |
| US11954813B2 (en) | Three-dimensional scene constructing method, apparatus and system, and storage medium | |
| CN111126359B (en) | High-definition image small target detection method based on self-encoder and YOLO algorithm | |
| CN116012817A (en) | Real-time panoramic parking space detection method and device based on double-network deep learning | |
| WO2020097840A1 (en) | Systems and methods for correcting a high-definition map based on detection of obstructing objects | |
| CN106971185B (en) | License plate positioning method and device based on full convolution network | |
| CN112507862B (en) | Vehicle orientation detection method and system based on multitasking convolutional neural network | |
| CN109741241B (en) | Fisheye image processing method, device, equipment and storage medium | |
| CN103049909B (en) | A kind of be focus with car plate exposure method | |
| CN113313047B (en) | Lane line detection method and system based on lane structure prior | |
| CN114663852A (en) | Method and device for constructing lane line graph, electronic equipment and readable storage medium | |
| CN114782865B (en) | Intersection vehicle positioning method and system based on multi-view and re-recognition | |
| CN116091574A (en) | 3D target detection method and system based on plane constraint and position constraint | |
| CN111881914B (en) | License plate character segmentation method and system based on self-learning threshold | |
| CN113361528A (en) | Multi-scale target detection method and system | |
| Gu et al. | Radar-enhanced image fusion-based object detection for autonomous driving | |
| CN114863096B (en) | Semantic map construction and positioning method and device for indoor parking lot | |
| CN116385477A (en) | Tower image registration method based on image segmentation | |
| CN115018926A (en) | Method, device and equipment for determining pitch angle of vehicle-mounted camera and storage medium | |
| Zheng et al. | A novel strategy for global lane detection based on key-point regression and multi-scale feature fusion | |
| CN116246252A (en) | Real-time panoramic parking space detection method, device and storage medium based on deep learning | |
| CN111553181A (en) | Vehicle-mounted camera semantic recognition method, system and device | |
| Sun et al. | Multi-Task Cross-Modality Attention-Fusion for 2D Object Detection | |
| CN113256736B (en) | Multi-camera visual SLAM method based on observability optimization | |
| CN116402871B (en) | Monocular distance measurement method and system based on scene parallel elements and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |