WO2023246720A1

WO2023246720A1 - Roadside parking detection method, roadside parking system, and electronic device

Info

Publication number: WO2023246720A1
Application number: PCT/CN2023/101179
Authority: WO
Inventors: 管文龙; 徐博文; 神克乐; 胡露露; 江璐; 龙一民; 陈新; 周浩; 荆碧晨
Original assignee: 阿里云计算有限公司
Priority date: 2022-06-20
Filing date: 2023-06-19
Publication date: 2023-12-28
Also published as: CN115171053A

Abstract

Provided in the embodiments of the present application are a roadside parking detection method, a roadside parking system, and an electronic device. In the technical solution provided in the embodiments of the present application, a plurality of machine learning tasks are fused into one multi-task detection model, and the multi-task detection model simultaneously outputs vehicle-related features such as a vehicle identifier and vehicle key point information which are associated with each other, thereby reducing the overhead of separately calling and executing a plurality of independent tasks; in addition, results such as the vehicle identifier and the vehicle key point information can be output in an associated manner by using the multi-task detection model during one instance of forward inference, thereby preventing a matching process after the execution of the plurality of independent tasks is completed. Therefore, the technical solution provided in the embodiments of the present application has high calculation efficiency and a relatively good level of detection accuracy; and the accurate detection of the key point information of the vehicle can allow the determination as to whether the vehicle meets the requirement for parking in a parking space, so as to perform corresponding handling on line pressing parking and cross-space parking of the vehicle, such that the level of intelligence of roadside parking management is further improved.

Description

Roadside parking detection method, system and electronic equipment

This application claims priority to the Chinese patent application filed with the China Patent Office on June 20, 2022, with application number 202210701367.0 and the application name "Roadside Parking Detection Method, System and Electronic Equipment", the entire content of which is incorporated by reference in in this application.

Technical field

The present application relates to the field of image processing technology, and in particular to a roadside parking detection method, system and electronic equipment.

Background technique

In recent years, the per capita car ownership in our country has continued to rise, and the problem of parking difficulties in public places has become increasingly prominent. Making full use of existing roadside open parking spaces is one of the effective means to alleviate this problem. Effective roadside parking management can reduce illegal parking and long-term parking.

In the past, parking management of open roadside parking spaces was mostly relied on manual labor, but there were problems such as chaotic charging management, low operational management efficiency, and high labor costs. With the development of visual perception technology, unattended roadside management solutions based on visual perception are expected to solve this problem. However, at present, when visual perception technology is applied in roadside parking management scenarios, there are still problems such as low vehicle detection efficiency and low detection accuracy.

Contents of the invention

The present application provides a roadside parking detection method, system and electronic device that solve the above problems or at least partially solve the above problems.

In one embodiment of the present application, a roadside parking detection method is provided. This method is suitable for roadside equipment. The specific methods include:

Collect images of vehicles parked on the roadside;

Use a multi-task detection model to detect the image to correlate and output vehicle identification and key point information;

Based on the key point information, determine whether the vehicle meets the parking requirements;

The determination result is associated with the vehicle identification as roadside parking data of the vehicle.

In another embodiment of the present application, a roadside parking detection method is provided. This method works for services terminal, the method includes:

Obtain training samples in the training set, where the training samples include a sample image, a first label and a second label; the first label is the vehicle identification of the vehicle in the sample image, and the second label is the sample Key point information of the vehicle in the picture;

Use the sample map as an input parameter of the multi-task detection model, and execute the multi-task detection model to associate and output a first result related to the vehicle identification and a second result related to the vehicle key point information;

Optimize parameters in the multi-task detection model according to the first result and the first label, and the second result and the second label;

After the multi-task detection model completes training, the trained multi-task detection model is sent to the roadside device, so that the roadside device uses the multi-task detection model to detect the collected images.

This application also provides an embodiment, a roadside parking system including:

The server is used to train the multi-task detection model using the training set;

Roadside equipment, connected to the server, is used to obtain the multi-task detection model that has completed training from the server; collect images of vehicles parked on the roadside; use the multi-task detection model to detect the images to Correlate and output vehicle identification and key point information; determine whether the vehicle meets the parking requirements based on the key point information;

In yet another embodiment of the present application, a roadside parking detection method is provided. This method can be applied to roadside equipment. The methods include:

Collect videos of roadside vehicles;

Use a multi-task detection model to detect image frames in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior;

Determine the length of time the vehicle is parked on the roadside based on the behavioral information;

Based on the key point information, determine whether the vehicle meets the parking requirements and obtain a determination result;

A parking bill is determined based on the vehicle identification, the parking duration and the determination result.

In an embodiment of the present application, an electronic device is also provided. The electronic device includes a memory and a processor; the memory is used to store one or more computer instructions. When the one or more computer instructions are executed by the processor, the road vehicle parking provided by the above embodiments can be realized. Steps in the detection method.

The technical solution provided by each embodiment of this application uses a multi-task detection model to detect the collected images of vehicles parked on the roadside to correlate and output vehicle identification and key point information; that is, each embodiment of this application uses multiple machine learning Tasks such as vehicle identification (such as license plate) detection tasks and vehicle key point detection tasks are integrated into one In the multi-task detection model; the multi-task detection model simultaneously outputs associated vehicle identification, vehicle key point information and other vehicle-related features, which reduces the overhead of calling and executing multiple independent tasks separately; in addition, the multi-task detection model can be used once The results such as vehicle identification and vehicle key point information are associated and output to the inference, which avoids the matching process after the execution of multiple independent tasks. It can be seen that the technical solution provided by the embodiment of the present application has high calculation efficiency and good detection accuracy. sex. With more accurate detection results, automated management of roadside parking can be realized. It can also determine whether the vehicle meets the parking requirements based on the key point information of the vehicle, so as to handle parking pressure, cross-space parking and other vehicles accordingly. .

Description of the drawings

In order to more clearly explain the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of a roadside parking system provided by an embodiment of the present application;

Figure 2 is a schematic flow chart of a roadside parking detection method provided by an embodiment of the present application;

Figure 3 is a schematic structural diagram of the principle of the multi-task detection model provided in an embodiment of the present application;

Figure 4 is a schematic diagram illustrating the image detection process provided by a multi-task detection model in an embodiment of the present application;

Figure 5 is a schematic flow chart of a roadside parking detection method provided by another embodiment of the present application;

Figure 6 is a schematic flow chart of a roadside parking detection method provided by another embodiment of the present application;

Figure 7 is a schematic structural diagram of a roadside parking detection device provided by an embodiment of the present application;

Figure 8 is a schematic structural diagram of a roadside parking detection device provided by another embodiment of the present application;

Figure 9 is a schematic structural diagram of a roadside parking detection device provided by another embodiment of the present application;

FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In order to enable those in the technical field to better understand the solution of the present application, the technical solution in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application.

Some of the processes described in the specification, claims, and above-mentioned drawings of this application include multiple operations that appear in a specific order. These operations may not be performed in the order in which they appear in this document or may be performed in parallel. The sequence numbers of operations, such as 101, 102, etc., are only used to distinguish different operations. The sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that descriptions such as “first” and “second” in this article are used to distinguish Different messages, devices, modules, etc. do not represent the order, nor does it limit "first" and "second" to be different types. In addition, the following embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the scope of protection of this application.

Figure 1 shows a schematic diagram of a roadside parking system. As shown in FIG. 1 , the roadside parking system in this embodiment may include but is not limited to: a server 11 and a roadside device 12 . Among them, the roadside equipment 12 may include but is not limited to: image acquisition device, storage device, networking device, power supply device, etc. Among them, the image collection equipment can be erected on the roadside using street lamp poles or other brackets to collect images, videos, etc. in the parking space area set along the roadside. The image acquisition device may have computing capabilities, such as image recognition capabilities, etc. Alternatively, the roadside equipment 12 further includes: a computing device connected to the image acquisition device for image recognition, data processing, etc. The server 11 may be a server deployed at the management center, a server cluster, a virtual server or a cloud server deployed on a physical machine, etc., which is not limited in this embodiment. In addition to the server and roadside equipment, the roadside parking system may also include: one or more clients 13, display screens 14, network video recorders 15 (Network Video Recorder, NVR, etc.), etc. Among them, the client 13 can be connected to the server 11 and can provide query services for users (such as staff), such as retrieving roadside parking images, roadside payment records, etc. from the server. The display screen 14 may be used to display images or videos of corresponding roadside parking areas collected by the roadside equipment. NVR is used to work with roadside equipment to complete the recording, storage and forwarding functions of images and videos.

The roadside device 12 is used to collect images or videos of the monitoring area, and has image or video analysis technology to implement image or video monitoring and target object (i.e., vehicle) tracking and identification functions. The roadside equipment 12 may also have a network connection function to transmit the images or videos collected by the image collection device and the processed data to the server 11 through the network. The server 11 receives the images, data, etc. uploaded by the roadside equipment at the monitoring points of each road section, and has a database function to store the images, data, etc. uploaded by the roadside equipment at the monitoring points of each road section, and provide them to, for example, time charging, staff Visual operations (such as calling and viewing through the client 13) and other management functions.

During specific implementation, the image collection equipment in the roadside equipment can be installed on the roadside, and its image collection perspective is aimed at the roadside parking space, and can capture images or videos of vehicles parked in its controlled parking spaces within the field of view. Roadside equipment can implement vehicle identity recognition functions, vehicle behavior recognition, etc. Among them, vehicle identity recognition requires identifying the vehicle license plate number that triggers evidence collection in the extracted key frames or photographed evidence collection images when the vehicle is parked or driven away, and the vehicle identification is obtained, which is used as the basis for generating parking amounts for parked vehicles in the background. Implement parking meter charging function. Vehicle behavior recognition requires identifying four behaviors during vehicle parking: starting to park, vehicle parking in, starting to drive out, and vehicle driving away. Among them, the purpose of judging the starting of parking and starting of driving out is to predict the parking and driving out behaviors before the vehicle parks into the parking space (or berth) or leaves the monitoring area, and is used to extract key frames or trigger the image acquisition device. Collect evidence to identify the vehicle's identity, so as to avoid the problem of the vehicle parking in the parking space, causing obstruction or driving out of the monitoring area, making it impossible to identify the vehicle's identity; judging the parking and driving behavior is to determine the time when the vehicle parked or left, which is used Calculate parking duration.

In fact, there are still some situations where parking spaces do not meet the requirements. For example, roadside vehicles do not park within the parking space marking area, which may affect the vehicles driving on the road; or roadside vehicles park across two parking spaces. parking spaces, which will result in a situation where one of the parking spaces cannot be parked. Therefore, it is necessary to increase the detection of vehicle parking conditions. Whether the vehicle parking meets the parking requirements (no line pressing, no crossing, etc.), this embodiment proposes key point information based on the vehicle (such as some pixels that represent the vehicle's occupancy, such as pixels where some wheels are in contact with the ground point) to make judgments.

That is, the roadside equipment 12 in this embodiment is used to collect images of vehicles parked on the roadside; use a multi-task detection model to detect the images to correlate and output vehicle identification and key point information; according to the key point information, Determine whether the vehicle meets the parking space requirements; associate the determination result with the vehicle identification as the roadside parking data of the vehicle.

Among them, the multi-task detection model utilized by the roadside device 12 can be deployed locally. The server 11 is responsible for training the multi-task detection model using the training set to continuously improve the capabilities of the multi-task detection model. The roadside device 12 may obtain the trained multi-task detection model from the server periodically or after receiving a model update instruction sent by the server.

In view of the characteristics of roadside parking scenarios, the embodiments of this application integrate vehicle key point information detection tasks, vehicle identification detection tasks, vehicle detection tasks, etc. into a multi-task detection model, thereby forming a new multi-task learning paradigm with performance and A better balance can be achieved in effect.

During specific implementation, the technical solution provided by the embodiment of the present application uses the anchor point of the vehicle detection frame as the regression starting point of the vehicle detection frame and vehicle key point information. In this positive sample allocation scheme, the vehicle key point information and vehicle detection The boxes are tied directly together. The vehicle key point information may be the key point information of the vehicle chassis, or further, the key point information corresponding to multiple wheels on the vehicle chassis. In the training sample allocation stage, the vehicle key point information and the vehicle detection frame are bound. Therefore, the training sample is used to train the multi-task detection model. The trained multi-task detection model can output the vehicle simultaneously in one forward inference. Detection frame, vehicle key point information and vehicle identification.

In the solution provided by this embodiment, the server 11 is used to train a multi-task detection model. The trained multi-task detection model can be deployed in the roadside device 12, that is, the computing tasks are decentralized to the edge device. Specifically, in this embodiment:

The server 11 is used to train the multi-task detection model using the training set.

The roadside device 12 is connected to the server 11 and is used to obtain the trained multi-task detection model from the server 11; collect images of vehicles parked on the roadside; and use the multi-task detection model to perform processing on the images. Detect and output the vehicle identification and key point information by association; determine whether the vehicle meets the parking requirements based on the key point information; associate the determination result with the vehicle identification as the roadside parking data of the vehicle.

The roadside parking data can be sent to the server 11 in real time, or can be stored locally on the roadside device for calculating the vehicle's parking bill or waiting for the server to pull it.

What needs to be added here is: the training process of the multi-task detection model will be explained in detail below.

Furthermore, the image detected by the roadside device 12 using the multi-task detection model mentioned above may be a frame in the video collected by the roadside device 12 . That is to say, in this embodiment, the roadside device 12 can also use the multi-task detection model to simultaneously detect the vehicle's behavior information, vehicle identification and key point information, so as to send the associated output information to the server, and the server can calculate the parking bill of the vehicle or Itself calculates the parking bill based on the information from the associated output. That is, the roadside equipment 12 described in this embodiment can also be used for:

Utilize the multi-task detection model to detect multiple frames of images in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior;

Determine the status of the vehicle based on the behavioral information;

When the state is a parking state, determine the parking starting time and associate the parking starting time with the vehicle identification;

After the state is a parking state, trigger the step of determining whether the vehicle meets the parking requirements based on the key point information;

When the state is the leaving state, the leaving time is determined, and the leaving time is associated with the vehicle identification.

Furthermore, the roadside equipment 12 can also be used to: send the parking start time, the determination result of whether the vehicle meets the parking requirements and the departure time to the server 11, and the server 11 based on the received data , determine the parking bill for that vehicle. Or the roadside device 12 is also used to: calculate the parking bill of the vehicle based on the parking start time, the determination result of whether the vehicle meets the parking requirements and the departure time; send the parking bill to the server, The server sends the parking bill to the corresponding user client.

That is, the server 11 in the system provided by this embodiment is also used to send a parking bill to the user client corresponding to the license plate identification, so that the user can pay the corresponding parking fee according to the parking bill. The parking fee may include: parking fee corresponding to the parking time, and additional fees to be paid when the parking space requirements are not met.

The technical solution provided by the embodiments of this application uses a multi-task detection model to detect the collected images of vehicles parked on the roadside to correlate and output vehicle identification and key point information; that is, each embodiment of this application uses multiple machine learning tasks , such as vehicle identification (such as license plate) detection tasks and vehicle key point detection tasks, etc. are integrated into one In the task detection model; the multi-task detection model simultaneously outputs associated vehicle identification, vehicle key point information and other vehicle-related features, which reduces the cost of calling and executing multiple independent tasks separately; in addition, because the multi-task detection model can be used in one forward During reasoning, results such as vehicle identification and vehicle key point information are associated and output, avoiding the matching process after the execution of multiple independent tasks. It can be seen that the technical solution provided by the embodiments of this application has high calculation efficiency and good detection accuracy. . With more accurate detection results, automated management of roadside parking can be realized. It can also determine whether the vehicle meets the parking requirements based on the key point information of the vehicle, so as to handle parking pressure, cross-space parking and other vehicles accordingly. .

In addition to the functions described above, the roadside equipment and the server in the above embodiments may also have other functions. The following will describe the corresponding functions of the roadside equipment and the server in the form of method steps.

FIG. 2 shows a schematic flowchart of a roadside parking detection method provided by an embodiment of the present application. The execution subject of the method provided in this embodiment may be a roadside device. Specifically, the method includes:

101. Collect images of vehicles parked on the roadside.

102. Use a multi-task detection model to detect the image to correlate and output vehicle identification and key point information.

103. Based on the key point information, determine whether the vehicle meets the parking requirements.

104. Associate the determination result with the vehicle identification as the roadside parking data of the vehicle.

In the above 101, the image may be an image collected after the vehicle is parked and parked, or a frame in a video collected by a roadside device.

In the above 102, the multi-task detection model can refer to the structure shown in Figure 3. Of course, this embodiment is not limited to the structure shown in the figure, and models corresponding to other network structures can also be used. As shown in FIG. 3 , the multi-task detection model may include: a feature extraction network 2 (or can be called a backbone network), a feature fusion layer 3 , a first branch network 4 and a second branch network 5 . Wherein, the first branch network is used to detect the vehicle identification of the vehicle in the image. The second branch network 5 can be used to detect key point information of the vehicle in the image. Further, as shown in FIG. 3 , the multi-task detection model may also include a third branch network 6 . The third branch network 6 is used to detect vehicles in the image and use detection frames to circle the vehicles.

An achievable technical solution is that the multi-task detection model in this embodiment can be based on a target detection neural network, such as the Yolo (You Only Look Once, an object recognition and positioning algorithm based on deep neural network) series, CenterNet series, etc., its overall architecture is shown in Figure 3. The detection head part of the multi-task detection model (i.e., the branch part after feature fusion layer 3) adds multiple new detection frame prediction bits and vehicle identification recognition prediction bits. Prediction bits are used to detect vehicle key point information. If the vehicle key point information includes key points corresponding to the four wheels of the vehicle chassis, 8 additional prediction bits can be added to detect the key points corresponding to the four wheels of the vehicle chassis.

In the above 103, the key point information may include: vehicle chassis key point information. More specifically, the key point information includes: key point information corresponding to multiple wheels on the vehicle chassis. For example, the contact point between the wheel and the ground is used as a key point. The key point information here may be: the pixel position information of the key point in the image, or the coordinate value in the coordinate system corresponding to the image acquisition device of the roadside equipment, etc. This embodiment is not limited to this. In addition to chassis key point information, the key point information may also include: feature points reflecting the outer contour of the vehicle, etc.

In an achievable technical solution, whether the vehicle meets the parking space requirements can be determined by analyzing the key point information and the parking space contour line at the vehicle parking position. For example, key point information includes: key point information corresponding to multiple wheels on the vehicle chassis. By analyzing the relative position relationship between the key point information and the parking space contour line at the vehicle parking position in the image, it is determined whether the vehicle parking meets the parking space requirements.

In the above 104, the determination result is associated with the vehicle identification as the roadside parking data of the vehicle. The roadside parking data can be uploaded to the server in real time; or the locally stored data can be stored in local batches and uploaded to the server. Roadside parking data can be used to calculate the vehicle's roadside parking bill, or used as a basis for corresponding processing of the vehicle's corresponding user when the vehicle does not meet the parking requirements. Further, when the vehicle does not meet the parking requirements, the image, the determination result and the vehicle identification are associated as the roadside parking data of the vehicle.

The technical solution provided by this embodiment uses a multi-task detection model to detect the collected images of vehicles parked on the roadside to correlate and output vehicle identification and key point information; that is, each embodiment of this application uses multiple machine learning tasks, For example, vehicle identification (such as license plate) detection tasks and vehicle key point detection tasks are integrated into a multi-task detection model; the multi-task detection model simultaneously outputs associated vehicle identification, vehicle key point information and other vehicle-related features, reducing the need for multiple independent The overhead of calling and executing tasks separately; in addition, because the multi-task detection model can be used to associate and output vehicle identification, vehicle key point information and other results in one forward reasoning, it avoids the matching process after the execution of multiple independent tasks; it can be seen that this The technical solution provided by the application embodiment has high calculation efficiency and good detection accuracy. With more accurate detection results, automated management of roadside parking can be realized. It can also determine whether the vehicle meets the parking requirements based on the key point information of the vehicle, so as to handle parking pressure, cross-space parking and other vehicles accordingly. .

Further, as shown in Figure 4, the multi-task detection model in this embodiment may include: feature extraction network 2 (or can be called a backbone network), feature fusion layer 3, first branch network 4 and second branch network 5 . in,

Feature extraction network 2 is used to extract features from the image 1 to obtain first feature information;

Feature fusion layer 3 is used to fuse the first feature information to obtain the second feature information;

The first branch network 4 is used to process the second feature information and output the vehicle identification;

The second branch network 5 is used to process the second feature information and output key point information.

Specifically, the key point information includes: anchor points of the detection frames corresponding to the vehicle in the image and key points of the vehicle chassis. Among them, the anchor point can be understood as: the center point of the predicted target (ie, the vehicle in the image). The detection frame and anchor point corresponding to the vehicle can be implemented by the third branch network. For example, the third branch network first predicts the anchor point of the prediction target in the image (ie, the vehicle in the image), and then predicts the length and width of the detection frame corresponding to the vehicle in the image based on the anchor point. That is, the multi-task detection model in this embodiment also includes a third branch network 6, as shown in Figure 4. The third branch network 6 is used to process the second feature information and output a detection frame 7 characterizing the vehicle position.

Referring to the example shown in Figure 4, the key point information of the vehicle includes: the anchor point 0 of the corresponding detection frame of the vehicle and the key points a, b, c and d corresponding to the four wheels on the vehicle chassis.

Further, the image in the above step 101 is a frame in the video. Correspondingly, the method provided in this embodiment may also include:

105. Use the multi-task detection model to detect multiple frames of images in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior;

Wherein, the behavior information includes: the anchor point of the detection frame corresponding to the vehicle detected from each frame image.

When it is determined that the vehicle is in a parking state based on the behavior information, the above step 103 is triggered to determine whether the vehicle meets the parking space requirements. That is, the method provided by the embodiments of this application may also include the following steps:

106. Determine the status of the vehicle based on the behavior information;

107. When the state is a parking state, determine the parking starting time and associate the parking starting time with the vehicle identification;

108. After the state is in the parking state, trigger the step of determining whether the vehicle meets the parking requirements based on the key point information (i.e. step 103);

109. When the state is the leaving state, determine the leaving time and associate the leaving time with the vehicle identification.

The above-mentioned parking start time, the determination result of whether the vehicle meets the parking requirements, and the departure time can be uploaded to the server or processed locally. Specifically, if processed locally, the method provided by the embodiment of this application may also include the following steps:

110. Calculate the parking fee for the vehicle corresponding to the vehicle logo based on the parking start time, the determination result of whether the vehicle meets the parking space requirements, and the departure time.

Further, the key point information includes: key points corresponding to multiple wheels on the vehicle chassis. Correspondingly, step 104 in the embodiment of this application "determine whether the vehicle meets the parking space requirements based on the key point information" can be implemented by the following steps:

1041. Obtain the parking space outline at the vehicle parking position in the image.

1042. Determine whether the vehicle meets the parking space requirements based on the parking space contour and key points corresponding to the multiple wheels of the vehicle chassis.

1043. When the determination result is that the parking space requirements are not met, the violation mode of the vehicle is determined based on the parking space outline and the key points corresponding to the multiple wheels of the vehicle chassis;

Wherein, the violation method includes at least one of the following: line pressing, crossing position.

In the above step 1042, it can be determined whether the vehicle meets the parking space requirements by analyzing the relative positional relationship between the parking space contour and the key points respectively corresponding to the multiple wheels of the vehicle chassis. For example, when the key point coordinates of at least one wheel of the vehicle chassis are located on the outer edge of the parking space outline extending along the road traveling direction, the vehicle pressure line is determined. If there is a horizontal edge line in the parking space contour line between the key point coordinates of the front wheel and the rear wheel key point coordinate of the vehicle chassis, which is used to separate two different parking spaces, the vehicle is determined to be in a straddle position. Of course, it is also possible that the vehicle is on the line and has a straddle position. These situations can be obtained by analyzing the relative positional relationship between the parking space contour and the key points corresponding to the multiple wheels of the vehicle chassis. This embodiment does not limit the specific analysis logic.

In the above step 1043, the determined violation mode can be added to the vehicle's roadside parking data to facilitate subsequent bill generation.

Figure 5 shows a schematic flowchart of a roadside parking detection method provided by another embodiment of the present application. As shown in the figure, the execution subject of the method provided in this embodiment may be a server. The methods include:

201. Obtain training samples in the training set, where the training samples include a sample image, a first label and a second label; the first label is the vehicle identification of the vehicle in the sample image, and the second label is the vehicle identification number of the vehicle in the sample image. Describe the key point information of the vehicle in the sample diagram.

202. Use the sample map as an input parameter of a multi-task detection model, and execute the multi-task detection model to associate and output a first result related to the vehicle identification and a second result related to the vehicle key point information.

203. Optimize the parameters in the multi-task detection model according to the first result and the first label, and the second result and the second label.

204. After the multi-task detection model completes training, send the trained multi-task detection model to the roadside device, so that the roadside device uses the multi-task detection model to detect the collected images.

In the above step 201, the training sample further includes a third label; the third label is the detection frame corresponding to the vehicle in the sample image; the key point information corresponding to the second label includes: the detection frame anchor. points and the chassis key points of the vehicle in the sample diagram; the result of executing the correlation output of the multi-task detection model also includes a third result related to the detection frame. Correspondingly, the above step 203 is specifically as follows:

According to the first result and the first label, the second result and the second label, and the third result and the third label, parameters in the multi-task detection model are optimized.

In a specific implementation, as shown in Figure 4, the multi-task detection model includes: feature extraction network 2, feature fusion layer 3, first branch network 4, second branch network 5 and third branch network 6; The input end of the first branch network 4 , the input end of the second branch network 5 and the input end of the third branch network 6 are all connected to the output end of the feature fusion layer 2 . Correspondingly, steps 202 and 203 "use the sample map as an input parameter of the multi-task detection model, execute the multi-task detection model to associate and output the first result, the second result and the third result; according to the first result and the first label, the second result and the second label, and the third result and the third label, optimizing the parameters in the multi-task detection model," may include the following steps:

S1. Input the sample image into the feature extraction network and output the first feature information;

S2. Input the first feature information into the feature fusion layer and output the second feature information;

S3. Input the second feature information into the first branch network and output the first result;

S4. Input the second feature information into the second branch network and output the second result;

S5. Input the second feature information into the third branch network and output the third result;

S6. According to the first result and the first label, the second result and the second label, and the third result and the third label, respectively extract the feature extraction network, the feature Optimize at least some parameters in the fusion layer, the first branch network, the second branch network and the third branch network.

Among them, when training the multi-task detection model in this embodiment, it is necessary to preprocess the labels of each sample image in the training set.

It should be noted here that the embodiments of this application do not specifically limit the loss function used in the optimization process of the multi-task detection model, nor the parameter optimization algorithm.

Figure 6 shows a schematic flowchart of a roadside parking detection method provided by yet another embodiment of the present application. The execution subject of the method provided in this embodiment may be a roadside device. Specifically, as shown in Figure 6, the method includes:

301. Collect videos of roadside vehicles.

302. Use a multi-task detection model to detect image frames in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior.

303. Determine the roadside parking duration of the vehicle based on the behavior information.

304. Based on the key point information, determine whether the vehicle meets the parking requirements and obtain a determination result.

305. Determine a parking bill based on the vehicle identification, the parking duration and the determination result.

Regarding the multi-task detection model in 302, how to determine whether the vehicle meets the parking space requirements in 304, etc., please refer to the corresponding content above, and will not be described in detail here.

In the above 303, the behavior information is the entire process of the vehicle from parking and warehousing behavior, parking to driving away. As above, it is necessary to first determine the state of the vehicle based on the behavior information. When the vehicle is in the parking state, the parking start time is determined; when the vehicle's behavior is the leaving state, the leaving time is determined. Calculate the length of time the vehicle will park on the roadside based on the departure time and parking start time.

In summary, the technical solution provided by each embodiment of the present application uses the anchor point of the vehicle corresponding detection frame as the regression starting point of the vehicle corresponding detection frame and vehicle key point information (such as key points corresponding to multiple wheels on the vehicle chassis), so that during training In the multi-task detection model, the vehicle key point information and the corresponding detection frame of the vehicle are directly bound together in the allocation plan for the positive samples in the training set. The multi-task detection model adopts a multi-task learning method. During the training process, vehicle key point information is added to the simple detection frame. The increase in features during the training process makes the multi-task detection model have good detection robustness.

In addition, the embodiment of the present application uses a multi-task detection model to simultaneously output vehicle identification, vehicle key point information, etc., reducing the overhead of multiple loop calls. The multi-task detection model binds vehicle key point information and vehicle corresponding detection frames in the positive sample allocation stage, eliminating subsequent pairing operations and improving the recall rate of vehicle key point information.

Multi-task detection models can train multiple interrelated tasks simultaneously. The multi-task detection model can use all data sets from different tasks during model training, which is equivalent to increasing the training samples of each task to a certain extent. It can not only improve the prediction performance, but also be used to solve the imbalance of data set samples. Or problems such as insufficient data samples. Secondly, the increase in the amount of sample data can also promote the development of the model in a more general direction, greatly improving the generalization of the model. In addition, using the same model to handle multiple different tasks simultaneously can reduce the overall time required to process these tasks.

The loss function of a multi-task model is the summation or other combination of multiple tasks, and based on assumptions about the model structure, several different regular terms are often added. Under the premise that the feature space of each task is isomorphic, each node of the output layer corresponds to the actual output of each task, and the error back propagation (Back Propagation, BP) algorithm can be used to train the network.

Figure 7 shows a schematic structural diagram of a roadside parking detection device provided by an embodiment of the present application. As shown in the figure, the device includes: a first acquisition module 21, a first detection module 22, a first determination module 23 and a data association module 24. Wherein, the first collection module 21 is used to collect images of vehicles parked on the roadside. The first detection module 22 is used to detect the image using a multi-task detection model to correlate and output vehicle identification and key point information. The first determination module 23 is used to determine whether the vehicle meets the parking space requirements based on the key point information. The data association module 24 is used to associate the determination result with the vehicle identification as the road of the vehicle. Side parking data.

Wherein, the multi-task detection model includes: a feature extraction network, a feature fusion layer, a first branch network, a second branch network and a third branch network. Wherein, the feature extraction network is used to extract features from the image to obtain first feature information. The feature fusion layer is used to fuse the first feature information to obtain the second feature information. The first branch network is used to process the second feature information and output a vehicle identification. The second branch network is used to process the second feature information and output key point information. The third branch network is used to process the second feature information and output a detection frame characterizing the vehicle position.

Further, the key point information may include: anchor points of the detection frames corresponding to the vehicle in the image and key points of the vehicle chassis.

Further, the image is a frame in a video. Correspondingly, the first detection module 22 is configured to use the multi-task detection model to detect multiple frames of images in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior. Wherein, the behavior information includes: the anchor point of the detection frame corresponding to the vehicle detected from each frame image.

Furthermore, the roadside parking detection device provided in this embodiment may also include a first determination module. Wherein, the first determination module is used to determine the state of the vehicle according to the behavior information; when the state is a parking state, determine the parking starting time, and combine the parking starting time with the vehicle identification Association; after the state is in the parking state, the first determination module is triggered to determine whether the vehicle meets the parking requirements according to the key point information; when the state is in the leaving state, the departure time is determined, and the The departure time is associated with the vehicle identification.

Furthermore, the first determination module is also used to calculate the parking fee for the vehicle corresponding to the vehicle logo based on the parking start time, the determination result of whether the vehicle meets the parking space requirements, and the departure time.

Further, the key point information includes: key points corresponding to multiple wheels on the vehicle chassis. Correspondingly, when determining whether the vehicle meets the parking requirements based on the key point information, the first determination module 23 is specifically used to:

Obtain the parking space contour line at the parking position of the vehicle in the image; determine whether the vehicle meets the parking space requirements based on the parking space contour line and key points corresponding to multiple wheels of the vehicle chassis; determine whether the vehicle meets the parking space requirements; When the result is that the parking space requirements are not met, the violation mode of the vehicle is determined based on the parking space contour and the key points respectively corresponding to the multiple wheels of the vehicle chassis; wherein the violation mode includes at least one of the following One type: line pressing and crossing position.

It should be noted here that the roadside parking detection device provided in this embodiment, in addition to the functions provided by each of the above modules, can also realize the functions corresponding to other part or all steps in the above corresponding method embodiments. For details, please refer to Please refer to the corresponding content of the above method embodiments, which will not be described again here.

Figure 8 shows a schematic structural diagram of a roadside parking detection device provided by another embodiment of the present application. As shown in the figure, the device includes: an acquisition module 31 , a training module 32 and a sending module 33 . Wherein, the acquisition module 31 is used to acquire training samples in the training set, where the training samples include a sample image, a first label and a second label; the first label is the vehicle identification of the vehicle in the sample image, and the The second label is the key point information of the vehicle in the sample image. The training module 32 is configured to use the sample graph as an input parameter of a multi-task detection model, and execute the multi-task detection model to associate and output a first result related to the vehicle identification and a second result related to the vehicle key point information. ; Optimize parameters in the multi-task detection model according to the first result and the first label, and the second result and the second label. The sending module 33 is configured to send the trained multi-task detection model to the roadside device after the multi-task detection model completes training, so that the roadside device uses the multi-task detection model to collect the images for inspection.

Further, the training sample also includes a third label; the third label is the detection frame corresponding to the vehicle in the sample image; the key point information corresponding to the second label includes: the anchor point of the detection frame and the The chassis key points of the vehicle in the sample diagram are shown; the result of executing the correlation output of the multi-task detection model also includes a third result related to the detection frame. Correspondingly, when optimizing the parameters in the multi-task detection model based on the first result and the first label, and the second result and the second label, the training module 32 is specifically used to:

Further, the multi-task detection model includes: a feature extraction network, a feature fusion layer, a first branch network, a second branch network and a third branch network; wherein, the input end of the first branch network, the second branch network The input terminal of and the input terminal of the third branch network are both connected to the output terminal of the feature fusion layer. Correspondingly, the training module 32 is specifically used to:

Input the sample image into the feature extraction network and output the first feature information;

Input the first feature information into the feature fusion layer and output the second feature information;

Input the second feature information into the first branch network and output the first result;

Input the second feature information into the second branch network and output the second result;

Input the second feature information into the third branch network and output the third result;

According to the first result and the first label, the second result and the second label, and the third result and the third label, the feature extraction network and the feature fusion layer are respectively , optimizing at least some parameters in the first branch network, the second branch network and the third branch network.

It should be noted here that the roadside parking detection device provided in this embodiment, in addition to the functions provided by each module mentioned above, can also realize the functions corresponding to other part or all steps in the corresponding method embodiments mentioned above. For details, see The corresponding contents of the above method embodiments will not be described again here.

Figure 9 shows a schematic structural diagram of a roadside parking detection device provided by yet another embodiment of the present application. As shown in the figure, the roadside parking detection device includes: a second collection module 41, a second detection module 42, a second determination module 43 and a second determination module 44. Wherein, the second collection module 41 is used to collect videos of roadside vehicles. The second detection module 42 is used to detect the image frames in the video using a multi-task detection model to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior. The second determination module 43 is configured to determine the roadside parking duration of the vehicle based on the behavior information. The second determination module 44 is used to determine whether the vehicle meets the parking requirements based on the key point information, and obtain a determination result. The second determination module is used to determine a parking bill based on the vehicle identification, the parking duration and the determination result.

Figure 10 shows a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 10 , the electronic device includes: a memory 51 and a processor 52 . Memory 51 may be configured to store various other data to support operations on the sensor. Examples of this data include instructions for any application or method operating on the sensor. Memory 51 may be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EEPROM), Programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The memory 51 is used to store one or more computer instructions;

The processor 52 is coupled to the memory 51 and is used to execute one or more computer instructions stored in the memory 51 to implement the steps in the roadside parking detection method provided by the above embodiments.

Further, as shown in FIG. 10 , the electronic device also includes: a communication component 53 , a power supply component 55 , a display 56 and other components. Only some components are schematically shown in FIG. 10 , which does not mean that the electronic device only includes the components shown in FIG. 10 .

Correspondingly, embodiments of the present application also provide a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, the steps or functions of the roadside parking detection method provided by the above embodiments can be implemented.

An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the processor can implement the steps or functions of the roadside parking detection method provided by the above embodiments.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims

A roadside parking detection method, characterized in that it is suitable for roadside equipment, and the method includes:

Collect images of vehicles parked on the roadside;

Use a multi-task detection model to detect the image to correlate and output vehicle identification and key point information;

Based on the key point information, determine whether the vehicle meets the parking requirements;

The determination result is associated with the vehicle identification as roadside parking data of the vehicle.
The method according to claim 1, characterized in that the multi-task detection model includes:

A feature extraction network, used to extract features from the image to obtain first feature information;

A feature fusion layer is used to fuse the first feature information to obtain the second feature information;

The first branch network is used to process the second feature information and output the vehicle identification;

The second branch network is used to process the second feature information and output key point information.
The method according to claim 2, characterized in that the multi-task detection model further includes:

The third branch network is used to process the second feature information and output a detection frame characterizing the position of the vehicle.
The method according to claim 3, characterized in that the key point information includes: anchor points of the detection frames corresponding to the vehicle in the image and key points of the vehicle chassis.
The method of claim 3, wherein the image is a frame in a video; and the method further includes:

Utilize the multi-task detection model to detect multiple frames of images in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior;

Wherein, the behavior information includes: the anchor point of the detection frame corresponding to the vehicle detected from each frame image.
The method according to claim 5, further comprising:

Determine the status of the vehicle based on the behavioral information;

When the state is a parking state, determine the parking starting time and associate the parking starting time with the vehicle identification;

After the state is a parking state, trigger the step of determining whether the vehicle meets the parking requirements based on the key point information;

When the state is the leaving state, the leaving time is determined, and the leaving time is associated with the vehicle identification.
The method according to claim 6, further comprising:

According to the parking start time, the determination result of whether the vehicle meets the parking requirements and the departure time to calculate the parking fee for the vehicle corresponding to the vehicle identification.
The method according to any one of claims 1 to 7, wherein the key point information includes: key points corresponding to multiple wheels on the vehicle chassis; and

Based on the key point information, determine whether the vehicle meets the parking requirements, including:

Obtain the parking space outline at the parking location of the vehicle in the image;

Determine whether the vehicle meets the parking space requirements according to the key points corresponding to the parking space contour and the multiple wheels of the vehicle chassis;

When the determination result is that the parking space requirements are not met, the violation mode of the vehicle is determined based on the parking space outline and the key points corresponding to the multiple wheels of the vehicle chassis;

Wherein, the violation method includes at least one of the following: line pressing, crossing position.
A roadside parking detection method, characterized in that it is suitable for the server, and the method includes:

Obtain training samples in the training set, where the training samples include a sample image, a first label and a second label; the first label is the vehicle identification of the vehicle in the sample image, and the second label is the sample Key point information of the vehicle in the picture;

Use the sample map as an input parameter of the multi-task detection model, and execute the multi-task detection model to associate and output a first result related to the vehicle identification and a second result related to the vehicle key point information;

Optimize parameters in the multi-task detection model according to the first result and the first label, and the second result and the second label;

After the multi-task detection model completes training, the trained multi-task detection model is sent to the roadside device, so that the roadside device uses the multi-task detection model to detect the collected images.
The method of claim 9, wherein the training sample further includes a third label; the third label is a detection frame corresponding to the vehicle in the sample image; and the key point information corresponding to the second label Including: the detection frame anchor point and the chassis key point of the vehicle in the sample map; the result of executing the correlation output of the multi-task detection model also includes a third result related to the detection frame; and

According to the first result and the first label, and the second result and the second label, optimizing the parameters in the multi-task detection model includes:

According to the first result and the first label, the second result and the second label, and the third result and the third label, parameters in the multi-task detection model are optimized.
The method according to claim 10, characterized in that the multi-task detection model includes: a feature extraction network, a feature fusion layer, a first branch network, a second branch network and a third branch network; wherein, the first The input end of the branch network, the input end of the second branch network and the input end of the third branch network are all connected to the output end of the feature fusion layer; and

The sample graph is used as an input parameter of the multi-task detection model, and the multi-task detection model is executed to associate and output the first result, the second result and the third result; according to the first result and the first label, the The second result and the second label and the third result and the third label are used to optimize parameters in the multi-task detection model, including:

Input the sample image into the feature extraction network and output the first feature information;

Input the first feature information into the feature fusion layer and output the second feature information;

Input the second feature information into the first branch network and output the first result;

Input the second feature information into the second branch network and output the second result;

Input the second feature information into the third branch network and output the third result;

According to the first result and the first label, the second result and the second label, and the third result and the third label, the feature extraction network and the feature fusion layer are respectively , optimizing at least some parameters in the first branch network, the second branch network and the third branch network.
A roadside parking system is characterized by including:

The server is used to train the multi-task detection model using the training set;

Roadside equipment, connected to the server, is used to obtain the multi-task detection model that has completed training from the server; collect images of vehicles parked on the roadside; use the multi-task detection model to detect the images to Correlate and output vehicle identification and key point information; determine whether the vehicle meets the parking requirements based on the key point information;

The determination result is associated with the vehicle identification as roadside parking data of the vehicle.
The system according to claim 12, wherein the image is a frame in a video; and the roadside device is also used for:

Utilize the multi-task detection model to detect multiple frames of images in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior;

Determine the status of the vehicle based on the behavioral information;

When the state is a parking state, determine the parking starting time and associate the parking starting time with the vehicle identification;

After the state is a parking state, trigger the step of determining whether the vehicle meets the parking requirements based on the key point information;

When the state is the leaving state, the leaving time is determined, and the leaving time is associated with the vehicle identification.
A roadside parking detection method, characterized in that it is suitable for roadside equipment, and the method includes:

Collect videos of roadside vehicles;

Use a multi-task detection model to detect image frames in the video to correlate and output behavioral information, vehicle identification and key point information reflecting the vehicle behavior;

Determine the length of time the vehicle is parked on the roadside based on the behavioral information;

Based on the key point information, determine whether the vehicle meets the parking requirements and obtain a determination result;

A parking bill is determined based on the vehicle identification, the parking duration and the determination result.
An electronic device, characterized in that it includes a memory and a processor; the memory is used to store one or more computer instructions, and when the one or more computer instructions are executed by the processor, the above claims 1 to 1 can be realized. The steps in the roadside parking detection method described in any one of 8, or the steps in the roadside parking detection method described in any one of claims 9 to 11, or the roadside parking detection described in claim 14 steps in the method.