WO2021135879A1 - Vehicle data monitoring method and apparatus, computer device, and storage medium - Google Patents

Abstract

A vehicle data monitoring method, comprising: obtaining a visual image, and identifying and extracting a vehicle and a traffic identifier in the visual image by means of a visual camera, so as to obtain image data (201); detecting vehicle data corresponding to the visual image by means of radar (202); performing mapping fusion on the image data and the vehicle data by means of information fusion, and obtaining fusion data (203); processing the fusion data by means of a target neural network, so as to monitor the vehicle data (204). Also provided are a vehicle data monitoring apparatus, a computer device, and a storage medium. The present invention increases the accuracy of information discrimination

Description

Method, device, computer equipment and storage medium for monitoring vehicle data

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on September 7, 2020 with the application number 202010931252.1 and the invention title "Methods, devices, computer equipment and storage media for monitoring vehicle data", the entire contents of which are incorporated by reference Incorporated in this application.

Technical field

This application relates to the field of artificial intelligence, and in particular to a method, device, computer equipment, and storage medium for monitoring vehicle data.

Background technique

Violation detection cameras are generally deployed in fixed locations such as bayonet and bridges, resulting in single violation detection capabilities, insufficient road violation monitoring coverage, and lack of supervision and punishment measures for a large number of violations during vehicle driving, thereby reducing the efficiency of urban road traffic. In the process of monitoring vehicle data, the inventor realized that the current product solution is still a single method based on visual analysis. Take the camera installed on a police vehicle as an example. Due to the complex road conditions, the current equipment can only complete the screening of suspected license plates. The judgment of road violations can only be done manually. Although some manufacturers have introduced deep learning technology to determine violations on the mobile terminal in specific scenarios, due to the complex road conditions, the two-dimensional image is due to inherent parallax and assistance. Lack of judgment information can easily lead to misjudgments.

Summary of the invention

The purpose of the embodiments of the present application is to provide a method, device, computer equipment, and storage medium for monitoring vehicle data, so as to improve the coverage of road violation detection.

In order to solve the above technical problems, an embodiment of the present application provides a method for monitoring vehicle data, which adopts the following technical solutions:

Obtain a visual image, identify and extract the vehicle and traffic identifiers in the visual image through a visual camera to obtain image data;

Detecting vehicle data corresponding to the visual image by radar;

Mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

The fusion data is processed through the target neural network to monitor vehicle data.

Further, the step of mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data specifically includes:

Obtaining the moving speed of the object according to the vehicle data, the object including at least a vehicle and a traffic identifier;

Obtaining the type of the object and the position of the object according to the image data;

Calculating the target average speed of the object according to the moving speed of the object;

Constructing a complete field of view based on the target average speed of the object, the type of the object, the position of the object, and the moving speed of the object;

Two-line interpolation of vehicle data in the complete field of view obtains the fusion data.

Further, the step of calculating the target average speed of the object according to the moving speed of the object specifically includes:

Obtain the lattice distribution in the radar field of view, and obtain the travel distance of the vehicle;

by

The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the distance traveled by the vehicle, n is the total number of points in the distance traveled by the vehicle, and V _avr is the distance traveled by the vehicle Average speed of travel;

by

Calculate the matching confidence, where σ is the matching confidence;

by

Normalize the matching confidence to obtain a normalized confidence;

If the normalized confidence is greater than a preset confidence threshold, then the V _{avr is} used as the target average speed.

Further, the step of acquiring the visual image, identifying and extracting the vehicle and the traffic identifier in the visual image through the visual camera to obtain the image data specifically includes:

Run a digital signal processing algorithm to process the visual image by means of direct storage access to obtain pre-processed visual data;

Recognizing vehicles and traffic identifiers in the pre-processed visual data through the recognition neural network;

After the step of processing the vehicle and the traffic identifier in the fusion data through the target neural network to monitor the vehicle data, the method includes:

If the vehicle violates regulations, detect and extract the license plate information of the vehicle and the violation information of the vehicle;

Output the license plate information of the vehicle and the violation information of the vehicle.

Further, the step of running a digital signal processing algorithm to process the visual image by means of direct storage access includes:

Count the number of the visual images input each time;

When the number of the visual image data input each time exceeds the image number threshold, the digital signal processing algorithm is run by using the thread pool of the image number threshold number;

When the number of visual image data input each time is less than or equal to the image number threshold, the digital signal processing algorithm is run according to the thread pool analysis of the number of visual image data input each time.

Further, before the step of counting the number of visual images input each time, the method further includes:

Arranging the number of the visual images inputted each time by the plurality of the statistics in time sequence to obtain an image sequence;

Obtain the maximum number of processed images N;

A target image is selected from the image sequence, the target image is taken as the first image, and N images are selected in order for input.

Further, after the step of recognizing the vehicle and the traffic identifier in the preprocessed visual data through the recognition neural network, the method further includes:

Obtain the vehicle that does not follow the traffic line in the image data, and obtain the violation information of the vehicle.

In order to solve the above technical problems, an embodiment of the present application also provides a device for monitoring vehicle data, which adopts the following technical solutions:

An acquisition module for acquiring a visual image, identifying and extracting a vehicle and a traffic identifier in the visual image through a visual camera to obtain image data;

The detection module is used to detect the vehicle data corresponding to the visual image through radar;

The fusion module is used for mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

The processing module is used to process the fused data through the target neural network to monitor vehicle data.

Further, the fusion module includes a speed acquisition sub-module, a position calculation sub-module, an average speed acquisition sub-module acquisition sub-module, a complete view field construction sub-module, and a two-line interpolation sub-module.

The speed acquisition sub-module is used to obtain the moving speed of the object according to the vehicle data, and the object includes at least a vehicle and a traffic identifier;

The position calculation sub-module is used to obtain the type of the object and the position of the object according to the image data;

The average speed acquisition sub-module is used to calculate the target average speed of the object according to the moving speed of the object;

The complete field of view construction sub-module is used to construct a complete field of view based on the target average speed of the object, the type of the object, the position of the object, and the moving speed of the object;

The two-line interpolation sub-module is used for two-line interpolation of vehicle data in the complete field of view to obtain the fusion data.

Further, the fusion module includes a distance acquisition sub-module, an average speed calculation sub-module, a confidence calculation sub-module, a normalization sub-module, and a threshold comparison sub-module.

The distance acquisition sub-module is used to acquire the lattice distribution in the radar field of view to obtain the travel distance of the vehicle;

The average speed calculation sub-module is used to pass

The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the distance traveled by the vehicle, n is the total number of points in the distance traveled by the vehicle, and V _avr is the distance traveled by the vehicle Average speed of travel;

The confidence calculation sub-module is used to pass

Calculate the matching confidence, where σ is the matching confidence;

The normalization sub-module is used to pass

Normalize the matching confidence to obtain a normalized confidence;

The threshold comparison sub-module is configured to use the V _avr as the target average speed if the normalized confidence is greater than a preset confidence threshold.

Further, the monitoring vehicle data device further includes a violation determination module, and the violation determination module includes an image processing sub-module, an identification sub-module, a detection sub-module, and a vehicle information output sub-module.

The image processing sub-module is used to run a digital signal processing algorithm to process the visual image by means of direct storage access to obtain pre-processed visual data;

The recognition sub-module is used to recognize the vehicle and the traffic identifier in the preprocessed visual data through the recognition neural network;

After the step of processing the fused data through the neural network, it specifically includes:

The detection sub-module is used to detect and extract the license plate information of the vehicle and the violation information of the vehicle if the vehicle violates regulations;

The vehicle information output sub-module is used to output the license plate information of the vehicle and the violation information of the vehicle.

Further, the violation judgment module includes a statistical sub-module and an image processing parallel sub-module.

The statistics sub-module is used to count the number of the visual images input each time;

The image processing parallel sub-module is configured to run the digital signal processing algorithm by using a thread pool of the threshold number of images when the number of input visual image data exceeds the threshold number of images each time;

When the number of visual image data input each time is less than or equal to the image number threshold, the digital signal processing algorithm is run according to the thread pool analysis of the number of visual image data input each time.

Further, the monitoring vehicle data device further includes a parallel processing module, and the parallel processing module includes an image sequence acquisition sub-module, a maximum processing number acquisition sub-module, and an image analysis sub-module.

The image sequence acquisition sub-module is configured to arrange the number of the visual images inputted each time by the plurality of the statistics in a time sequence to obtain an image sequence;

The maximum processing number acquisition sub-module is used to acquire the maximum processing image number N;

The image analysis sub-module is used for selecting a target image from the image sequence, taking the target image as the first image, and sequentially selecting N images for input.

Further, the monitoring vehicle data device further includes an overspeed module, and the overspeed module includes a violation information acquisition submodule.

The violation information acquisition sub-module is used to acquire the vehicles that do not follow the traffic line in the image data, and obtain the violation information of the vehicles.

In order to solve the above technical problems, the embodiments of the present application also provide a computer device, which adopts the following technical solutions:

A computer device includes at least one connected processor, a memory, and a network interface, wherein the memory is used to store computer readable instructions, and the processor is used to call the computer readable instructions in the memory to execute the following The steps of the method for monitoring vehicle data:

Obtain a visual image, identify and extract the vehicle and traffic identifiers in the visual image through a visual camera to obtain image data;

Detecting vehicle data corresponding to the visual image by radar;

Mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

The fusion data is processed through the target neural network to monitor vehicle data.

In order to solve the above technical problems, the embodiments of the present application also provide a computer-readable storage medium, which adopts the following technical solutions:

A computer-readable storage medium having computer-readable instructions stored thereon, and when the computer-readable instructions are executed by a processor, the steps of the method for monitoring vehicle data as described below are implemented:

Obtain a visual image, identify and extract the vehicle and traffic identifiers in the visual image through a visual camera to obtain image data;

Detecting vehicle data corresponding to the visual image by radar;

Mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

The fusion data is processed through the target neural network to monitor vehicle data.

Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

This application introduces a method of monitoring vehicle data in the mobile violation detection platform. In the violation detection process, through the introduction of millimeter wave radar and matching with vehicle image data, the vehicle position, speed and three-dimensional space location information of transportation facilities are effectively obtained , Increase the dimension of information acquisition, improve the accuracy of the terminal algorithm, greatly reduce the possibility of misjudgment of violations, and reduce the labor cost of later violation reviews.

Description of the drawings

In order to explain the solution in this application more clearly, the following will briefly introduce the drawings used in the description of the embodiments of the application. Obviously, the drawings in the following description are some embodiments of the application. Ordinary technicians can obtain other drawings based on these drawings without creative work.

Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

Fig. 2-1 A flowchart of an embodiment of a method for monitoring vehicle data according to the present application;

Figure 2-2 A schematic diagram of the installation of a radar and a vision camera in the method for monitoring vehicle data according to the present application;

Figure 2-3 A schematic diagram of the mapping and fusion of radar monitoring data and visual images according to the method for monitoring vehicle data according to the present application;

Fig. 3 is a schematic structural diagram of an embodiment of a device for monitoring vehicle data according to the present application;

Fig. 4 is a schematic structural diagram of an embodiment of a computer device according to the present application.

Detailed ways

Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the technical field of the application; the terms used in the specification of the application herein are only for describing specific embodiments. The purpose is not to limit the application; the terms "including" and "having" in the specification and claims of the application and the above-mentioned description of the drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings.

As shown in FIG. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.

The user can use the terminal devices 101, 102, and 103 to interact with the server 105 through the network 104 to receive or send messages and so on. Various communication client applications, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social platform software, may be installed on the terminal devices 101, 102, and 103.

The terminal devices 101, 102, and 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic Video experts compress standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image experts compress standard audio layer 4) players, laptop portable computers and desktop computers, etc.

The server 105 may be a server that provides various services, for example, a background server that provides support for pages displayed on the terminal devices 101, 102, and 103.

It should be noted that the method for monitoring vehicle data provided by the embodiments of the present application is generally executed by a server/terminal device. Accordingly, the device for monitoring vehicle data is generally set in the server/terminal device.

It should be understood that the numbers of terminal devices, networks, and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks, and servers according to implementation needs.

With continued reference to FIG. 2, a flowchart of an embodiment of the method for monitoring vehicle data according to the present application is shown. The method for monitoring vehicle data includes the following steps:

Step 201: Obtain a visual image, identify and extract the vehicle and the traffic identifier in the visual image through the visual camera to obtain image data.

In this embodiment, the electronic device (such as the server/terminal device shown in FIG. 1) on which the method for monitoring vehicle data runs can be calibrated by receiving a user request through a wired connection or a wireless connection. It should be pointed out that the above-mentioned wireless connection methods can include, but are not limited to, 3G/4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future .

In this implementation, the visual image is obtained through a radar and a vision camera, and the vision camera and the radar antenna are installed on the same plane of the device for detection of the vehicle ahead and the road identifier. As shown in Figure 2-2, the circle on the left is the camera lens, and the millimeter wave radar antenna is on the right. The vehicle and traffic identifiers are detected by training the recognition neural network for detecting the target. Complete the detection and extraction of vehicle and traffic identifiers through the vision system, including vehicle position, solid lines, dashed lines, yellow solid lines, diversion lines, etc. After completing the recognition of the visual image, the image data is obtained. The detection of road edges, the collection of vehicle position and vehicle speed are completed by millimeter wave radar. The vision system is mainly composed of multiple vision cameras.

Step 202: Detect vehicle data corresponding to the visual image by radar.

In this embodiment, the vehicle data includes vehicle position and vehicle speed. The detection and extraction of vehicles and traffic identifiers are completed by the vision system, including vehicle position, solid lines, dashed lines, yellow solid lines, diversion lines, etc. The detection of road edges, the collection of vehicle position and vehicle speed are completed by millimeter wave radar. Among them, what the radar detects is not a plane point, but a point with a spatial structure, and the vision system is mainly composed of multiple vision cameras. Generally, in vehicle monitoring, in order to monitor the surrounding road conditions in all directions without blind spots, a single camera and radar cannot be completed, so multiple components are required to complete all-round data monitoring.

Step 203: Perform mapping and fusion on the image data and the vehicle data through information fusion to obtain fusion data.

In this embodiment, as shown in Figure 2-3, the left side is the radar field of view, the right side is the image field of view, the radar is used to detect objects and moving speed in the field, and the image field of view is used to detect the type and location of the object. By combining the two fields of view, vehicle and object discrimination and relative movement speed discrimination are performed. Since the ratios of the two viewing areas are usually not completely the same, it is necessary to sacrifice a part of the detection ability through cropping to ensure the consistency of the viewing area ratio; finally, the radar detection area is expanded and mapped to the image area through the bilinear interpolation method (Usually the radar field of view will be smaller than the image field of view), the interpolation process only processes the position of the radar detection point, but does not change the size of the changed point and the speed information. Obtain the lattice distribution in the radar field of view, obtain the travel distance of the vehicle, and pass

The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the travel distance of the vehicle, n is the total number of points in the travel distance _{of the vehicle, and V avr} is the travel distance of the vehicle The average speed; thus completing the fusion of k data.

Step 204: Process the fusion data through the target neural network to monitor vehicle data.

In this embodiment, three aspects of perception, decision-making and control are included in the driving field. Perception refers to how to parse out the information of the surrounding environment through the input of cameras and other sensors, such as which obstacles, the speed and distance of the obstacles, the width and curvature of the road, etc. Control refers to how to achieve this goal by adjusting the mechanical parameters of the car when there is a goal, such as turning 30 degrees to the right. When the target neural network has the fusion data, it can be used to judge whether the car has violations.

This application introduces a method of monitoring vehicle data in a mobile violation detection platform. Through the introduction of millimeter wave radar and matching with vehicle image data, the vehicle position, speed, and three-dimensional spatial location information of transportation facilities are effectively obtained, and the information acquisition dimension is increased. , Improve the accuracy of the terminal algorithm, greatly reduce the possibility of misjudgment of violations, and reduce the labor cost of later violation reviews.

In some optional implementation manners, the moving speed of the object is obtained according to the vehicle data, and the object includes at least a vehicle and a traffic identifier;

Obtaining the type of the object and the position of the object according to the image data;

Calculating the target average speed of the object according to the moving speed of the object;

Based on the target average speed of the object, the type of the object, the position of the object, and the moving speed of the object, a complete field of view is constructed.

Two-line interpolation of vehicle data in the complete field of view obtains the fusion data.

In the foregoing embodiment, the recognition neural network model is used to perform vehicle detection and traffic sign segmentation respectively. The deep learning algorithm refers to how to parse out the information of the surrounding environment through the input of the camera and other sensors, such as which obstacles, the speed and distance of the obstacles, the width and curvature of the road, etc. The vehicle speed measurement is realized by the speed measurement radar array. The confidence level S(σ) and S(σ) of the vehicle speed are calculated to determine whether the vehicle speed is less than a value to screen whether the measured vehicle speed is credible. The speed and position of the object are also the same, and passed It is calibrated with the camera data to obtain the speed distribution of all vehicles within the viewing angle. The two-line difference is applied. When the number of radar points appearing in a certain place is too dense due to too many noise points, which affects the judgment, some points near a certain point are eliminated.

In some optional implementation manners, the step of calculating the target average speed of the object according to the moving speed of the object specifically includes:

Obtain the lattice distribution in the radar field of view, and obtain the travel distance of the vehicle;

by

The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the distance traveled by the vehicle, n is the total number of points in the distance traveled by the vehicle, and V _avr is the distance traveled by the vehicle Average speed of travel;

by

Calculate the matching confidence, where σ is the matching confidence;

by

Normalize the matching confidence to obtain a normalized confidence;

If the normalized confidence is greater than a preset confidence threshold, then the V _{avr is} used as the target average speed.

In the above embodiment, in the radar field of view, the vehicle is composed of a series of lattices, and the visually collected vehicle position is matched with the lattice to determine the distance and speed of the vehicle. S(σ)<S, the data is considered unreliable, and the detected vehicle data is deleted. Through this method, the credibility of the data is detected, and the misjudgment caused by the wrong detection result is prevented. When the moving speed of car A is 10m/s, then according to the location of nearby traffic signs, nearby cyclists, non-moving cars, the location of various obstacles, and the moving speed of cyclists and pedestrians, a complete vehicle and pedestrian can be constructed. Image data.

In some optional implementation manners, before the step of processing the fused data through a neural network, the method further includes:

Run a digital signal processing algorithm (Digital Signal Process, DSP) to process the visual image by means of Direct Memory Access (DMA) to obtain preprocessed image data;

Recognizing vehicles and traffic identifiers in the pre-processed visual data through the recognition neural network;

After the step of processing the vehicle and the traffic identifier in the fusion data through the target neural network to determine whether the vehicle is speeding and violating regulations, the method includes:

If the vehicle violates regulations, detect and extract the license plate information of the vehicle and the violation information of the vehicle;

Output the license plate information of the vehicle and the violation information of the vehicle.

In the above embodiments, the image signal processing sub-module is mainly used to output signal processing sub-modules to the front-end image sensor to match image sensors of different manufacturers. Through the DMA technology, the camera data is distributed to the traditional video processing sub-module and the algorithm analysis sub-module at the same time. Through the pipeline processing of the collected pictures, the requirements for hardware computing power and power consumption are reduced. The programmable logic gate array (Field Programmable Gate Arra, FPGA) parallelizes the advantages of processing deep learning algorithms, and uses the central processing unit (CPU) to complete logic operations. Under the condition of limited cost and increased power consumption, the existing Semiconductor technology effectively improves the terminal algorithm processing capacity and image processing throughput, realizes the real-time completion of road violation detection algorithms on embedded devices, and transmits the extracted structured information and violation proofs to the traffic management bureau server, which greatly reduces System bandwidth requirements, while reducing the labor costs of later violation reviews.

In some optional implementation manners, the step of processing the visual image by running a digital signal processing algorithm by means of direct storage access specifically includes:

Count the number of the visual images input each time;

When the number of the visual image data input each time exceeds the image number threshold, the digital signal processing algorithm is run by using the thread pool of the image number threshold number;

When the number of visual image data input each time is less than or equal to the image number threshold, the digital signal processing algorithm is run according to the thread pool analysis of the number of visual image data input each time.

In the above embodiment, since preprocessing, vehicle detection/segmentation, and violation identification/license plate detection are all time-consuming operations, after the encoding process is completed, the digital signal processing algorithm is run in a multi-threaded manner to complete the encoding flow through BUF1, BUF2... Data buffering, through the "sampling control" sub-module to realize data multi-stage pipeline processing, the number of BUFx (xth buffer area) is designed according to the actual algorithm execution time, by setting a certain number of buffer areas, that is, the number of image buffers. Zone, to ensure the feasibility of parallel operation of the algorithm.

In some optional implementation manners, before the step of counting the number of visual images input each time, the method further includes:

Arranging the number of the visual images inputted each time by the plurality of the statistics in time sequence to obtain an image sequence;

Obtain the maximum number of processed images N;

A target image is selected from the image sequence, the target image is taken as the first image, and N images are selected in order for input.

In the above-mentioned embodiment, the violation judgment algorithm outputs before, during, after, and after the violation and close-up pictures of the vehicle, extracts the detection result of the license plate, and outputs the time point of the violation judgment, and extracts the time within t before the violation occurred (t according to the violation evidence Request design) Violation information, and save it in the violation video file, and send the data. Taking 10 seconds per time and 10 frames per second as an example, 100 frames of images are taken as input each time, and the first 10 frames are deleted for the images in the next second, and the frame number of the last frame is moved back by 10 frames.

In some optional implementation manners, after the step of identifying the vehicle and the traffic identifier in the pre-processed image data through the first neural network, the method further includes:

Obtain the vehicle that does not follow the traffic line in the image data, and obtain the violation information of the vehicle.

In the above-mentioned embodiment, in the target neural network model, the vehicle detection data and traffic sign segmentation information are used to determine whether the vehicle is not traveling along the traffic line. If it is determined that the vehicle is not traveling along the traveling line, it is determined that the vehicle is in violation.

In some optional implementations, the vehicle data corresponding to the visual image detected by the radar may be stored in a block.

The blockchain referred to in this application is a new application mode of computer technology such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information for verification. The validity of the information (anti-counterfeiting) and the generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through computer-readable instructions, which can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above-mentioned method embodiments. Among them, the aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

It should be understood that although the various steps in the flowchart of the drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless explicitly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the order of execution is also It is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

With further reference to FIG. 3, as an implementation of the method shown in FIG. 2-1, this application provides an embodiment of a device for monitoring vehicle data. The device embodiment corresponds to the method embodiment shown in FIG. The device can be specifically applied to various electronic devices.

As shown in FIG. 3, the device 300 for monitoring vehicle data in this embodiment includes: an acquisition module 301, a detection module 302, a fusion module 303, and a processing module 304. among them:

The fetching module 301 is used to obtain a visual image, and use a visual camera to identify and extract the vehicle and traffic identifiers in the visual image to obtain image data;

The detection module 302 is configured to detect vehicle data corresponding to the visual image through radar;

The fusion module 303 is used for mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

The processing module 304 is configured to process the fusion data through the target neural network to determine whether the vehicle is speeding and violating regulations.

In some optional implementations of this embodiment, the fusion module of the aforementioned fusion module includes a speed acquisition sub-module, a position calculation sub-module, an average speed acquisition sub-module acquisition sub-module, a complete view field construction sub-module, and a two-line interpolation Submodule:

The speed acquisition sub-module is used to obtain the moving speed of the object according to the vehicle data, and the object includes at least a vehicle and a traffic identifier;

The position calculation sub-module is used to obtain the type of the object and the position of the object according to the image data;

The average speed acquisition sub-module is used to calculate the target average speed of the object according to the moving speed of the object;

The complete field of view construction sub-module is used to construct a complete field of view based on the target average speed of the object, the type of the object, the position of the object, and the moving speed of the object;

The two-line interpolation sub-module is used for two-line interpolation of vehicle data in the complete field of view to obtain the fusion data.

In some optional implementations of this embodiment, the aforementioned fusion module includes a distance acquisition submodule, an average speed calculation submodule, a confidence calculation submodule, a normalization submodule, and a threshold comparison submodule:

The distance acquisition sub-module is used to acquire the lattice distribution in the radar field of view to obtain the travel distance of the vehicle;

The average speed calculation sub-module is used to pass

The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the distance traveled by the vehicle, n is the total number of points in the distance traveled by the vehicle, and V _avr is the distance traveled by the vehicle Average speed of travel;

The confidence calculation sub-module is used to pass

Calculate the matching confidence, where σ is the matching confidence;

The normalization sub-module is used to pass

Normalize the matching confidence to obtain a normalized confidence;

The threshold comparison sub-module is configured to use the V _avr as the target average speed if the normalized confidence is greater than a preset confidence threshold.

In some optional implementations of this embodiment, the above-mentioned device 300 further includes: the monitoring vehicle data device further includes a violation determination module, the monitoring vehicle data device further includes a violation determination module, and the violation determination module includes: Image processing sub-module, recognition sub-module, detection sub-module and vehicle information output sub-module:

The image processing sub-module is used to run a digital signal processing algorithm to process the visual image by means of direct storage access to obtain pre-processed visual data;

The recognition sub-module is used to recognize the vehicle and the traffic identifier in the preprocessed visual data through the recognition neural network;

After the step of processing the fused data through the neural network, it specifically includes:

The detection sub-module is used to detect and extract the license plate information of the vehicle and the violation information of the vehicle if the vehicle violates regulations;

The vehicle information output sub-module is used to output the license plate information of the vehicle and the violation information of the vehicle.

In some optional implementation manners of this embodiment, the violation judgment module includes a statistics sub-module and an image processing parallel sub-module:

The statistics sub-module is used to count the number of the visual images input each time;

The image processing parallel sub-module is configured to run the digital signal processing algorithm by using a thread pool of the threshold number of images when the number of input visual image data exceeds the threshold number of images each time;

When the number of visual image data input each time is less than or equal to the image number threshold, the digital signal processing algorithm is run according to the thread pool analysis of the number of visual image data input each time.

In some optional implementations of this embodiment, the above-mentioned device 300 further includes: a parallel processing sub-module, the device for monitoring vehicle data further includes a parallel processing module, the device for monitoring vehicle data further includes a parallel processing module, and the parallel processing The modules include image sequence acquisition sub-module, maximum processing number acquisition sub-module, and image analysis sub-module are also used for:

The image sequence acquisition sub-module is configured to arrange the number of the visual images inputted each time by the plurality of the statistics in a time sequence to obtain an image sequence;

The maximum processing number acquisition sub-module is used to acquire the maximum processing image number N;

The image analysis sub-module is used for selecting a target image from the image sequence, taking the target image as the first image, and sequentially selecting N images for input.

In some optional implementation manners of this embodiment, the above-mentioned apparatus 300 further includes: a violation judging submodule, and the overspeeding module includes a violation information acquiring submodule:

The violation information acquisition sub-module is used to acquire the vehicles that do not follow the traffic line in the image data, and obtain the violation information of the vehicles.

In order to solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 4 for details. FIG. 4 is a block diagram of the basic structure of the computer device in this embodiment.

The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are connected to each other in communication via a system bus. It should be pointed out that the figure only shows the computer device 4 with components 41-43, but it should be understood that it is not required to implement all the shown components, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, a dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Processor (Digital Signal Processor, DSP), embedded equipment, etc.

The computer device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touch panel, or a voice control device.

The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static memory Random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 6, a smart media card (SMC), a secure digital (Secure Digital, SD) card, Flash Card, etc. Of course, the memory 41 may also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed in the computer device 4, such as computer-readable instructions for monitoring vehicle data. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

The processor 42 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to run computer-readable instructions or process data stored in the memory 41, for example, run the computer-readable instructions of the aforementioned method for monitoring vehicle data.

The network interface 43 may include a wireless network interface or a wired network interface, and the network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

This application also provides another implementation manner, that is, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and the computer-readable storage medium may be non-volatile. Volatile, or volatile, the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor, so that the at least one processor can execute as described above The steps of the method of monitoring vehicle data.

Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.

Obviously, the embodiments described above are only a part of the embodiments of the present application, rather than all of the embodiments. The drawings show preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these examples is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it is still possible for those skilled in the art to modify the technical solutions described in each of the foregoing specific embodiments, or equivalently replace some of the technical features. . All equivalent structures made by using the contents of the description and drawings of this application, directly or indirectly used in other related technical fields, are similarly within the scope of patent protection of this application.

Claims (20)

1. A method for monitoring vehicle data includes the following steps:

   Obtain a visual image, identify and extract the vehicle and traffic identifiers in the visual image through a visual camera to obtain image data;

   Detecting vehicle data corresponding to the visual image by radar;

   Mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

   The fusion data is processed through the target neural network to monitor vehicle data.
2. The method for monitoring vehicle data according to claim 1, wherein the step of mapping and fusing the image data and the vehicle data through information fusion to obtain the fusion data specifically comprises:

   Obtaining the moving speed of the object according to the vehicle data, the object including at least a vehicle and a traffic identifier;

   Obtaining the type of the object and the position of the object according to the image data;

   Calculating the target average speed of the object according to the moving speed of the object;

   Constructing a complete field of view based on the target average speed of the object, the type of the object, the position of the object, and the moving speed of the object;

   Two-line interpolation of vehicle data in the complete field of view obtains the fusion data.
3. The method for monitoring vehicle data according to claim 2, wherein the step of calculating the target average speed of the object according to the moving speed of the object specifically comprises:

   Obtain the lattice distribution in the radar field of view, and obtain the travel distance of the vehicle;

   by

   

   The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the distance traveled by the vehicle, n is the total number of points in the distance traveled by the vehicle, and V _avr is the distance traveled by the vehicle Average speed of travel;

   by

   

   Calculate the matching confidence, where σ is the matching confidence;

   by

   

   Normalize the matching confidence to obtain a normalized confidence;

   If the normalized confidence is greater than a preset confidence threshold, then the V _{avr is} used as the target average speed.
4. The method for monitoring vehicle data according to any one of claims 1 to 3, wherein the step of obtaining the visual image, identifying and extracting the vehicle and the traffic identifier in the visual image by a visual camera, to obtain the image data is specific include:

   Run a digital signal processing algorithm to process the visual image by means of direct storage access to obtain pre-processed visual data;

   Recognizing vehicles and traffic identifiers in the pre-processed visual data through the recognition neural network;

   After the step of processing the vehicle and the traffic identifier in the fusion data through the target neural network to monitor the vehicle data, the method includes:

   If the vehicle violates regulations, detect and extract the license plate information of the vehicle and the violation information of the vehicle;

   Output the license plate information of the vehicle and the violation information of the vehicle.
5. The method for monitoring vehicle data according to claim 4, wherein the step of processing the visual image by running a digital signal processing algorithm by means of direct storage access comprises:

   Count the number of the visual images input each time;

   When the number of the visual image data input each time exceeds the image number threshold, the digital signal processing algorithm is run by using the thread pool of the image number threshold number;

   When the number of visual image data input each time is less than or equal to the image number threshold, the digital signal processing algorithm is run according to the thread pool analysis of the number of visual image data input each time.
6. The method for monitoring vehicle data according to claim 5, wherein before the step of counting the number of visual images input each time, the method further comprises:

   Arranging the number of the visual images inputted each time by the plurality of the statistics in time sequence to obtain an image sequence;

   Obtain the maximum number of processed images N;

   A target image is selected from the image sequence, the target image is taken as the first image, and N images are selected in order for input.
7. The method for monitoring vehicle data according to claim 4, wherein after the step of recognizing the vehicle and the traffic identifier in the pre-processed visual data through the recognition neural network, the method further comprises:

   Obtain the vehicle that does not follow the traffic line in the image data, and obtain the violation information of the vehicle.
8. A device for monitoring vehicle data includes:

   An acquisition module for acquiring a visual image, identifying and extracting a vehicle and a traffic identifier in the visual image through a visual camera to obtain image data;

   The detection module is used to detect the vehicle data corresponding to the visual image through radar;

   The fusion module is used for mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

   The processing module is used to process the fused data through the target neural network to monitor vehicle data.
9. A computer device includes a memory and a processor, wherein computer readable instructions are stored in the memory, and when the processor executes the computer readable instructions, the following steps of the method for monitoring vehicle data are implemented:

   Obtain a visual image, identify and extract the vehicle and traffic identifiers in the visual image through a visual camera to obtain image data;

   Detecting vehicle data corresponding to the visual image by radar;

   Mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

   The fusion data is processed through the target neural network to monitor vehicle data.
10. The computer device according to claim 9, wherein the step of mapping and fusing the image data and the vehicle data through information fusion to obtain the fusion data specifically comprises:

    Obtaining the moving speed of the object according to the vehicle data, the object including at least a vehicle and a traffic identifier;

    Obtaining the type of the object and the position of the object according to the image data;

    Calculating the target average speed of the object according to the moving speed of the object;

    Constructing a complete field of view based on the target average speed of the object, the type of the object, the position of the object, and the moving speed of the object;

    Two-line interpolation of vehicle data in the complete field of view obtains the fusion data.
11. 11. The computer device according to claim 10, wherein the step of calculating the target average speed of the object according to the moving speed of the object specifically comprises:

    Obtain the lattice distribution in the radar field of view, and obtain the travel distance of the vehicle;

    by

    

    The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the distance traveled by the vehicle, n is the total number of points in the distance traveled by the vehicle, and V _avr is the distance traveled by the vehicle Average speed of travel;

    by

    

    Calculate the matching confidence, where σ is the matching confidence;

    by

    

    Normalize the matching confidence to obtain a normalized confidence;

    If the normalized confidence is greater than a preset confidence threshold, then the V _{avr is} used as the target average speed.
12. The computer device according to any one of claims 9-11, wherein the step of obtaining a visual image, identifying and extracting a vehicle and a traffic identifier in the visual image by a visual camera, to obtain image data specifically comprises:

    Run a digital signal processing algorithm to process the visual image by means of direct storage access to obtain pre-processed visual data;

    Recognizing vehicles and traffic identifiers in the pre-processed visual data through the recognition neural network;

    After the step of processing the vehicle and the traffic identifier in the fusion data through the target neural network to monitor the vehicle data, the method includes:

    If the vehicle violates regulations, detect and extract the license plate information of the vehicle and the violation information of the vehicle;

    Output the license plate information of the vehicle and the violation information of the vehicle.
13. The computer device according to claim 12, wherein the step of processing the visual image by running a digital signal processing algorithm by means of direct storage access comprises:

    Count the number of the visual images input each time;

    When the number of the visual image data input each time exceeds the image number threshold, the digital signal processing algorithm is run by using the thread pool of the image number threshold number;

    When the number of visual image data input each time is less than or equal to the image number threshold, the digital signal processing algorithm is run according to the thread pool analysis of the number of visual image data input each time.
14. The computer device according to claim 13, wherein before the step of counting the number of the visual images input each time, the method further comprises:

    Arranging the number of the visual images inputted each time by the plurality of the statistics in time sequence to obtain an image sequence;

    Obtain the maximum number of processed images N;

    A target image is selected from the image sequence, the target image is taken as the first image, and N images are selected in order for input.
15. A computer-readable storage medium, wherein computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 7 is realized. Steps of the method of monitoring vehicle data.

    Obtain a visual image, identify and extract the vehicle and traffic identifiers in the visual image through a visual camera to obtain image data;

    Detecting vehicle data corresponding to the visual image by radar;

    Mapping and fusing the image data and the vehicle data through information fusion to obtain fusion data;

    The fusion data is processed through the target neural network to monitor vehicle data.
16. The computer-readable storage medium according to claim 15, wherein the step of mapping and fusing the image data and the vehicle data through information fusion to obtain the fused data specifically comprises:

    Obtaining the moving speed of the object according to the vehicle data, the object including at least a vehicle and a traffic identifier;

    Obtaining the type of the object and the position of the object according to the image data;

    Calculating the target average speed of the object according to the moving speed of the object;

    Constructing a complete field of view based on the target average speed of the object, the type of the object, the position of the object, and the moving speed of the object;

    Two-line interpolation of vehicle data in the complete field of view obtains the fusion data.
17. The computer-readable storage medium according to claim 16, wherein the step of calculating the target average speed of the object according to the moving speed of the object specifically comprises:

    Obtain the lattice distribution in the radar field of view, and obtain the travel distance of the vehicle;

    by

    

    The average speed of the vehicle is calculated, where v _k is the speed of the k-th point in the distance traveled by the vehicle, n is the total number of points in the distance traveled by the vehicle, and V _avr is the distance traveled by the vehicle Average speed of travel;

    by

    

    Calculate the matching confidence, where σ is the matching confidence;

    by

    

    Normalize the matching confidence to obtain a normalized confidence;

    If the normalized confidence is greater than a preset confidence threshold, then the V _{avr is} used as the target average speed.
18. The computer-readable storage medium according to any one of claims 15-17, wherein the step of obtaining a visual image, identifying and extracting a vehicle and a traffic identifier in the visual image by a visual camera, to obtain image data Specifically:

    Run a digital signal processing algorithm to process the visual image by means of direct storage access to obtain pre-processed visual data;

    Recognizing vehicles and traffic identifiers in the pre-processed visual data through the recognition neural network;

    After the step of processing the vehicle and the traffic identifier in the fusion data through the target neural network to monitor the vehicle data, the method includes:

    If the vehicle violates regulations, detect and extract the license plate information of the vehicle and the violation information of the vehicle;

    Output the license plate information of the vehicle and the violation information of the vehicle.
19. 18. The computer-readable storage medium according to claim 18, wherein the step of running a digital signal processing algorithm to process the visual image by means of direct storage access comprises:

    Count the number of the visual images input each time;

    When the number of the visual image data input each time exceeds the image number threshold, the digital signal processing algorithm is run by using the thread pool of the image number threshold number;

    When the number of visual image data input each time is less than or equal to the image number threshold, the digital signal processing algorithm is run according to the thread pool analysis of the number of visual image data input each time.
20. 18. The computer-readable storage medium according to claim 19, wherein before the step of counting the number of visual images input each time, the method further comprises:

    Arranging the number of the visual images inputted each time by the plurality of the statistics in time sequence to obtain an image sequence;

    Obtain the maximum number of processed images N;

    A target image is selected from the image sequence, the target image is taken as the first image, and N images are selected in order for input.

Images